How do I view the Node (Aggregator/Leaf) tracelogs in a Kubernetes environment?
To view the tracelogs of a node, locate the corresponding Pod and run:
kubectl logs <POD NAME> node
Replace <POD NAME> with the complete name of the Pod. For example, in this test Kubernetes cluster, the Pods are labeled as such:
> kubectl get pods
NAME READY STATUS RESTARTS AGE
memsql-operator-55b8c88878-vt4pd 1/1 Running 2 5d21h
node-memsql-cluster-aggregator-0 2/2 Running 4 6d2h
node-memsql-cluster-leaf-ag1-0 2/2 Running 4 6d2h
node-memsql-cluster-leaf-ag2-0 2/2 Running 4 6d2h
node-memsql-cluster-master-0 2/2 Running 4 6d2h
If I wanted to check the logs for node-memsql-cluster-leaf-ag1-0 (one of the leaf nodes), then I would run:
kubectl logs node-memsql-cluster-leaf-ag1-0 node
Each Pod also contains an exporter container that also has logging. Typically you would primarily want to check the Master (MA) for exporter information. You can do this by changing node to the exporter in the above command:
> kubectl logs node-memsql-cluster-master-0 exporter
time="2022-05-31T16:53:06Z" level=info msg="Arguments: " source="memsql_exporter.go:621"
time="2022-05-31T16:53:06Z" level=info msg="Starting memsql_exporter (version=7.8.4, branch=, revision=edc23b1)" source="memsql_exporter.go:636"
You can also go within the PersistentVolume resource itself to collect the logs. You can check the files themselves by going into the instance directory within the PV storage itself.
Alternatively, you can attach to a shell in the Pod to view the files (in this instance, I'm going within the node-memsql-cluster-leaf-ag2-0 Pod).
kubectl exec -it node-memsql-cluster-leaf-ag2-0 -- bash
From there, you can change the directory to /var/lib/memsql, and you will see the PV is mounted there. Node files are held within the instance directory.
You may utilize the kubectl cp command to extract files from within the Pod. For example (using the same Pod from the most recent example):
kubectl cp default/node-memsql-cluster-leaf-ag2-0:/var/lib/memsql/instance/tracelogs/memsql.log /home/ubuntu/memsql.log
The above command will cp (copy) the tracelogs in a Pod in the default namespace named node-memsql-cluster-leaf-ag2-0 to the local drive. The tracelogs are located at /var/lib/memsql/instance/tracelogs/memsql.log within the Pod and will be copied to /home/ubuntu/memsql.log on the local drive.
Also: If you take a cluster report to send to us it will also include all the logging we need.