{question}
How do I manually trigger a Core Dump to send to SingleStore Support?
{question}
{answer}
During SingleStore Support investigations, it may sometimes be necessary to collect a core dump from a SingleStore node that is experiencing an issue not easily diagnosable through standard methods such as logs or the information schema. This guide explains how to manually trigger a core dump and outlines the information that should be provided to Support via the support ticket.
Verify the core limit of the memsqld process you will core:
1) First, the source of truth for checking if a core file is enabled is checking
/proc/pid/limits for the parent memsqld process (SingleStore daemon).For example (lowest pid is the parent pid):
ubuntu@kafka:~$ ps aux | grep memsqld | grep -v memsqld_safe | awk '{print $2,$11}'
606026 grep
2363703 /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld <------- parent pid
2363704 /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld <------- child pid
2) Run
cat /proc/PID/limits | grep core:ubuntu@kafka:~$ cat /proc/2363703/limits | grep core Max core file size unlimited unlimited bytes
3) If core size is not set to unlimited, you can set it to unlimited with
prlimit --core=unlimited --pid=PID. That way, the core dump isn't truncated upon creation (making it unusable for debugging).ubuntu@kafka:~$ cat /proc/2363703/limits | grep core Max core file size 0 0 bytes ubuntu@kafka:~$ sudo prlimit --core=unlimited --pid=2363703 ubuntu@kafka:~$ cat /proc/2363703/limits | grep core Max core file size unlimited unlimited bytes
Note:
prlimit will not allow you to exceed the user's hard limit. The hard limit is typically defined in /etc/security/limits.confFor example:
ubuntu@kafka:~$ cat /etc/security/limits.conf * soft core unlimited * hard core unlimited <------------------------ memsql soft NOFILE 1024000 memsql hard NOFILE 1024000 memsql soft nproc 128000 memsql hard nproc 128000 memsql - nice -10
4) On the Singlestore side, confirm
core_file is set to 1 or "on", for the node in question that you are going to core: ubuntu@kafka:~$ memsql -pmemsql -P 3307 -e 'select @@core_file' singlestore-client: [Warning] Using a password on the command line interface can be insecure. +-------------+ | @@core_file | +-------------+ | 1 | +-------------+
Force the memsqld to create a core file:
Warning: these instructions will force
memsqld to crash! Confirm there is enough disk space available to accommodate the creation of the core file. The core file on disk can be approximately up to the size of theTotal_server_memory of the memsql node. 1) The
-ABRTkill signal is used to force a memsqld process to core dump, for example (using the example parent PID from Step 1) of the last section of this guide)- ubuntu@kafka:~$ sudo kill -ABRT 23637032) Using the
-ABRTsignal on the memsqld process will create a file called core file with the naming convention and path specified in /proc/sys/kernel/core_patternFor example:
ubuntu@kafka:/tmp$ cat /proc/sys/kernel/core_pattern /tmp/core-%e-%s-%u-%g-%p-%t ubuntu@kafka:/tmp$ ls | grep core core-memsqld-6-113-119-2363703-1692124203
3) Compress and send the core dump to the SingleStore Support team.
Collect the lib files used to analyze the core dump:
1) The below command creates a directory in your
/tmpdirectory with all the necessary lib files to analyze the core dump and attach this to the ticket:mkdir /tmp/libs && ldd [path_to_memsqld] | awk 'NF == 4 {print $3}; NF == 2 {print $1}' | grep -v "linux-vdso" | xargs -I '{}' cp {} -t /tmp/libs/
You can find
path_to_memsqld by checking ps. In the below example, the path is /opt/singlestoredb-server-8.1.15-4660571fdf/memsqldubuntu@kafka:~$ ps aux | grep memsqld | grep -v memsqld_safe | awk '{print $2,$11}'
606026 grep
2363703 /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld
2363704 /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld
2) Compress and send the lib directory created to the SingleStore Support team.
{answer}