Articles in this section

See more

Locating core dumps and adjusting core file size limits when using systemd-coredump

{question}

How to find core dumps and modify core file size limits when using systemd-coredump?

{question}

{answer}

This guide explains how to locate core dumps and adjust core file size limits for systems utilizing systemd-coredump.

Confirm if systemd-coredump is managing core files

 
1) To determine if systemd-coredump is managing core files, you can look up kernel.core_pattern in sysctl_stdout to verify systemd-coredump Is there:
 
ubuntu@kafka:~$ sudo sysctl -A | grep pattern 
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
 

Verify Core Dump File Size Limits

 
1) If the core_pattern specifies systemd-coredump, check /etc/systemd/coredump.conf and modify the values to a value at least as high as the value of what you have configured for maximum_memory for the respective node in Singlestore, so that core dumps aren't truncated (and unusable). 
 
[Coredump] 
ProcessSizeMax=50G
ExternalSizeMax=50G 
JournalSizeMax=100G

 

Locate the core file(s)

1) Run coredumpctl list
ubuntu@kafka:~$ coredumpctl list
TIME PID UID GID SIG COREFILE EXE SIZE
Wed 2023-12-06 18:48:37 UTC 71347 114 121 SIGABRT inaccessible /opt/singlestoredb-server-8.1.30-e0a67e68e5/memsqld n/a
 
2) Run coredumpctl info PID where PID is what's listed from the coredumpctl list output to confirm where the core file is. In this example, it's in /var/lib/systemd/coredump/:
ubuntu@kafka:~$ coredumpctl info 71347
       PID: 71347 (memsqld)
       UID: 114 (memsql)
       GID: 121 (memsql)
    Signal: 6 (ABRT)
 Timestamp: Wed 2023-12-06 18:48:29 UTC (13min ago)
 Command Line: /opt/singlestoredb-server-8.1.30-e0a67e68e5/memsqld --defaults-file ...
 Executable: /opt/singlestoredb-server-8.1.30-e0a67e68e5/memsqld
 Control Group: /user.slice/user-1000.slice/session-2251.scope
          Unit: session-2251.scope
         Slice: user-1000.slice
       Session: 2251
    Owner UID: 1000 (ubuntu)
      Boot ID: 1126edb4aa674b0ba092468a03efee65
   Machine ID: 970e8353a4364205a8b6964379f8418b
      Hostname: ip-172-31-31-120
       Storage: /var/lib/systemd/coredump/core.memsqld.114.1126edb4aa674b0ba092468a03efee65.71347.1701888509000000.zst
       Message: Process 71347 (memsqld) of user 114 dumped core.

Found module /opt/singlestoredb-server-8.1.30-e0a67e68e5/memsqld with build-id: ...
Stack trace of thread 71347:
#0 0x00007fb6ae318dbf n/a (n/a + 0x0)
 
3) Can see the core file is here:
ubuntu@kafka:~$ cd /var/lib/systemd/coredump/
ubuntu@kafka:/var/lib/systemd/coredump$ ls
core.memsqld.114.1126edb4aa674b0ba092468a03efee65.71347.1701888509000000.zst

 

4) Send Singlestore Support the core dump (compressed file ending in .zst)

ubuntu@kafka:/var/lib/systemd/coredump$ sudo zstd -d core.memsqld.114.1126edb4aa674b0ba092468a03efee65.71347.1701888509000000.zst
core.memsqld.114.1126edb4aa674b0ba092468a03efee65.71347.1701888509000000.zst: 2147483648 bytes

ubuntu@ip-172-31-31-120:/var/lib/systemd/coredump$ ls -larth
total 448M
drwxr-xr-x 10 root root 4.0K Nov  8  2022 ..
-rw-r-----  1 root root  39M Dec  6 18:48 core.memsqld.114.1126edb4aa674b0ba092468a03efee65.71347.1701888509000000.zst
-rw-r-----  1 root root 2.0G Dec  6 18:48 core.memsqld.114.1126edb4aa674b0ba092468a03efee65.71347.1701888509000000      <-----------------
drwxr-xr-x  2 root root 4.0K Dec  6 19:02 .

Collect the lib files used to analyze the core dump:

 
1) The below command creates a directory in your /tmpdirectory with all the necessary lib files to analyze the core dump and attach this to the ticket:
mkdir /tmp/libs && ldd [path_to_memsqld] | awk 'NF == 4 {print $3}; NF == 2 {print $1}' | grep -v "linux-vdso" | xargs -I '{}' cp {} -t /tmp/libs/
 
You can find path_to_memsqld out by checking ps. In the below example, the path is /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld
ubuntu@kafka:~$ ps aux | grep memsqld | grep -v memsqld_safe | awk '{print $2,$11}'
606026 grep
2363703 /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld
2363704 /opt/singlestoredb-server-8.1.15-4660571fdf/memsqld

 

2) Compress and send the lib directory created to the SingleStore Support team.

{answer}

Related articles