{question}
How can I generate a Flame Graph?
{question}
{answer}
Flame graphs are a way of visualizing CPU stacks to help understand CPU usage. On Linux, we use perf (perf events) to sample CPU stacks and post-process them using the FlameGraph scripts.
Steps to generate a Flame Graph while collecting perf data:
1) Install Flamegraph scripts and perf
Perf should already be installed as a part of the Linux tools. If not, install the correct Linux tools package for your host. You might need:
linux-tools-common
linux-tools-generic
Flamegraph scripts:
Clone the git repository to somewhere like /tmp:
$ git clone https://github.com/brendangregg/FlameGraph.git /tmp
2) Enable Debug Mode
This ensures we get usable stack traces inside the generated code. If it is not enabled, then most elements will be labeled as "unknown", which does not help identify what was using the CPU.
On the node you are concerned about CPU usage in, run the command below. It can be set at runtime like this and does not require node restart.
set global debug_mode = on;
3) Drop the plancache
Function names are only included in the plan and displayed in the flamegraph if debug_mode is enabled when the plan is compiled. For all previously compiled plans, this is not the case, so they need to be recompiled.
Run these commands below in a database connection to the node in question to get the current values and disable plancaching for both the in-memory and on-disk plancaches:
select @@disk_plan_expiration_minutes;
select @@plan_expiration_minutes;
set global disk_plan_expiration_minutes = 0;
set global plan_expiration_minutes = 0;
Wait 2 minutes to ensure that the plan garbage collector has run. To confirm the plancache size has dropped, check how many plans are there with the command below. If any queries are currently running (including some internal database operations), then the count will be slightly greater than 0.
select count(*) from information_schema.plancache;
Set the values back to their defaults (or any custom values that were set originally):
set global disk_plan_expiration_minutes = 20160;
set global plan_expiration_minutes = 720;
4) Run the workload.
Run the performance test as in step 4) of the original plan.
5) Record Performance:
While the performance test is running, collect 10 seconds of stacks via perf with the command below. The parent_memsqld_PID_here is the memsqld process (not memsqld_safe) that you can find it using `ps -ef | grep memsqld`.
sudo perf record -F 997 --call-graph lbr -p parent_memsqld_PID_here -g -o perf.data -- sleep 10
If this command hangs for some reason please use:
sudo perf record -F 997 -a -g -o perf.data -- sleep 10
Note: The flag --call-graph lbr requires a Haswell or newer CPU. If the CPU is not compatible, you may need to run it with this value instead: --call-graph dwarf
6) Adjust Permissions:
The owner of the perf.data file and the /tmp/perf-<pid>.stack must match. Change the owner of the files generated in /tmp/ to the desired user:
sudo chown $USER perf.data
sudo chown $USER /tmp/perf*
sudo chown $USER /tmp/_MEI*
7) Generate a Flamegraph:
Use perf and the FlameGraph scripts to generate an SVG flamegraph:
perf script -i perf.data | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > flamegraph.svg
8) Disable Debug Mode:
Now that the test is over disable debug mode, we are not storing this extra labeling information, which can consume disk storage if left enabled for a long time.
set global debug_mode = off;
Send the flamegraph.svg file that is generated to the SingleStore support team. If you collected it on an aggregator and a leaf, send both flamegraph.svg files (one for each host).
{answer}