Procedure to determine which query crashed a leaf node

{question}

How do I determine which query crashed a leaf node?

{question}

{answer}

Troubleshooting Steps

1) Navigate to the tracelogs directory of the leaf node that crashed.

• By default, the tracelogs are located in /<path-to-memsql-install-directory>/tracelogs. Within that directory, there is a file called memsql.log.

2) Grep the memsql.log file for traces such asPrintCallStackor RegisterCrashReport.

• Above these traces, you'll see a distributed query (leaf query) printed to the tracelogs, for example:

query: USING `testdb_1`::`temp_1_23456_7890`  AGGREGATOR_ID = 5, CONNECTION_ID = 12345/*!90624 , HAS_TEMP_TABLES = 0*/ /*!90623 , XACT_ID = (123456789 )*/, AGGREGATOR_PLAN_HASH = 'some_query_plan_here_c123456789' /*!90621 OBJECT()*/ /*!90623 RG_POOL 0*/ SELECT WITH(binary_serialization=1, binary_serialization_internal=1)
...
[libmemtrack.so (0x7fc896f1b32b)] backtrace 0x3B
[memsqld (0x332e755)] PrintCallStack(_IO_FILE*) 0x25
[memsqld (0x16dd620)] RegisterCrashReport 0xD0

• Then you will want to take note of the AGGREGATOR_ID and AGGREGATOR_PLAN_HASH. In the above example, these are AGGREGATOR_ID = 5and AGGREGATOR_PLAN_HASH = 'some_query_plan_here_c123456789'.
• Note: In some cases, when nodes crash, the AGGREGATOR_PLAN_HASHis not stated in the crash stack. If this is the case, it is likely that SingleStore support will need to dig deeper into the logging and analyze core dumps to determine the reason behind node crashes. See the Collecting information to analyze a core file and the Filing SingleStore Support Tickets knowledge base articles for more details on what to send to the SingleStore Support team for analysis.

3) If the plan for the offending query still exists, you will be able to find the query text that was executed on the aggregator that resulted in the leaf node crashing by consulting INFORMATION_SCHEMA.MV_PLANCACHE.

• Run the following query on the master aggregator to see the aggregator query text of the offending plan:

singlestore> SELECT QUERY_TEXT FROM INFORMATION_SCHEMA.MV_PLANCACHE WHERE NODE_ID = 5 AND ACTIVITY_NAME = 'Select_t_e0f283cfcd2823ab';
+----------------------------+
| QUERY_TEXT                 |
+----------------------------+
| offending query text here  |
+----------------------------+
1 row in set (0.01 sec)

4) Alternatively you can find the query text in INFORMATION_SCHEMA.MV_QUERIES  by filtering on activity_name where you'll see the de-parameterized text of the query in the query_text.

{answer}