Using trace signal
The trace signal gadget is used to trace system signals received by the pods.
First, we need to create one pod for us to play with:
$ kubectl run debian --image debian:latest sleep inf
You can now use the gadget, but output will be empty:
$ kubectl gadget trace signal NODE NAMESPACE POD CONTAINER PID COMM SIGNAL TPID RET
Indeed, it is waiting for signals to be sent.
So, in another terminal,
exec the container and send one signal:
$ kubectl exec -ti debian -- sh -c 'sleep 3 & kill -kill $!'
Go back to the first terminal and see:
NODE NAMESPACE POD CONTAINER PID COMM SIGNAL TPID RET minikube default debian debian 129484 sh SIGKILL 129491 0 minikube default debian debian 129484 sh SIGHUP 129491 0 minikube default debian debian 129484 sh SIGHUP 129484 0
The first line corresponds to
kill sending signal
You can also use this gadget to trace when processes die with segmentation fault.
In the other terminal,
exec the container with the following:
$ kubectl exec -ti debian -- bash # We first need to install python2. # apt update && apt install -qy python2 # We can now generate a segfault. # python2.7 -c "exec'()'*7**6"
Now, go back to the first terminal and see that
SIGSEGV was sent to python:
minikube default debian debian 142244 python2.7 SIGSEGV 142244 0
Congratulations! You reached the end of this guide! You can now delete the pod you created:
$ kubectl delete pod debian pod "debian" deleted
Start the gadget on a terminal.
$ sudo local-gadget trace signal -c test-trace-signal
Run a container and run sleep in the background, then will it:
$ docker run -it --rm --name test-trace-signal busybox /bin/sh / # sleep 100 & / # echo $! 7 / # kill -kill $! / # exit
The gadget will show that sh killed a process.
$ sudo local-gadget trace signal -c test-trace-signal CONTAINER PID COMM SIGNAL TPID RET test-trace-signal 11131 sh SIGKILL 11162 0 test-trace-signal 11131 sh SIGHUP 11131 0
Restricting output to certain PID, signals or failed to send the signals
With the following option, you can restrict the output:
--pidonly prints events where a signal is sent by the given PID.
--signalonly prints events where the given signal is sent.
-f/--failed-onlyonly prints events where signal failed to be delivered.
-k/--kill-onlyonly prints events where signal was sent by using kill syscall.
For example, this command will only print failed attempts to send
SIGKILL by PID
42 which were initiated by calling kill syscall:
$ kubectl gadget -k -f --pid 42 --signal SIGKILL
Note that, with
--signal you can use the name of the signal (e.g.
SIGKILL) or its integer value (e.g. 9).