local-gadget
Inspektor Gadget relies on the Kubernetes API server to work. However, there are
some cases
where it is necessary, or preferred, to trace
containers without passing through Kubernetes. In such scenarios, you can use
the local-gadget
tool, as it allows you to collect insights from the nodes to
debug your Kubernetes containers without relying on Kubernetes itself, but on
the container runtimes. It is important to remark that local-gadget
can also
be used to trace containers that were not created via Kubernetes.
Some characteristics of the local-gadget
:
- It uses eBPF as its underlying core technology.
- Enriches the collected data with the Kubernetes metadata.
- Easy to install as it is a single binary (statically linked).
The architecture of local-gadget
is described in the main
architecture
document.
Use cases
- In a Kubernetes environment, when the Kubernetes API server is not working properly, we cannot deploy Inspektor Gadget. Therefore, we still need a way to debug the containers of the cluster.
- In some cases, you might have root SSH access to the Kubernetes nodes of a
cluster, but not to the
kubeconfig
. - If you are implementing an application that needs to get insights from the
Kubernetes node, you could include the
local-gadget
binary in your container image, and your app simply execs it. In such a case, it is suggested to use the JSON output format to ease the parsing. - Outside a Kubernetes environment, for observing and debugging standalone containers.
Installation
The instruction to install local-gadget
are available in the main
installation
guide.
Usage
Currently, the local-gadget
can trace containers managed by Docker regardless
of whether they were created via Kubernetes or not. In addition, it can also
use the CRI to trace containers managed by containerd and CRI-O, meaning only
the ones created via Kubernetes. Support for non-Kubernetes containers with
containerd is coming, see issue
#734
.
By default, the local-gadget
will try to communicate with the Docker Engine
API and the CRI API of containerd and CRI-O:
$ docker run -d --name myContainer nginx:1.21
95b814bb82b9e30dd935b03d04a7b00b6978ce018a6f55d6a9c7a824b31ec6b5
$ sudo local-gadget list-containers
WARN[0000] Runtime enricher (cri-o): couldn't get current containers
RUNTIME ID NAME
containerd 7766d32caded4 calico-kube-controllers
containerd 2e3e4968b456f calico-node
containerd d3be7741b94ff coredns
containerd e7be3e4dc1bb4 coredns
containerd fb4fe41921f30 etcd
containerd 136e7944d2077 kube-apiserver
containerd ad8709a2c2ded kube-controller-manager
containerd 66cf05654a47f kube-proxy
containerd a68bed42aa6b2 kube-scheduler
docker 95b814bb82b9e myContainer
This output shows the containers local-gadget
retrieved from Docker and
containerd, while the warning message tells us that local-gadget
tried to
communicate with CRI-O but couldn’t. In this case, it was because CRI-O was not
running in the system where we executed the test. However, it could also happen
if local-gadget
uses a different UNIX socket path to communicate with the
runtimes. To check which paths local-gadget
is using, you can use the --help
flag:
$ sudo local-gadget list-containers --help
List all containers
Usage:
local-gadget list-containers [flags]
Flags:
...
--containerd-socketpath string containerd CRI Unix socket path (default "/run/containerd/containerd.sock")
--crio-socketpath string CRI-O CRI Unix socket path (default "/run/crio/crio.sock")
--docker-socketpath string Docker Engine API Unix socket path (default "/run/docker.sock")
-r, --runtimes string Container runtimes to be used separated by comma. Supported values are: docker, containerd, cri-o (default "docker,containerd,cri-o")
...
If needed, we can also specify the runtimes to be used and their UNIX socket path:
$ sudo local-gadget list-containers --runtimes docker --docker-socketpath /some/path/docker.sock
RUNTIME ID NAME
docker 95b814bb82b9e myContainer
Common features
Notice that most of the commands support the following features even if, for simplicity, they are not demonstrated in each command guide:
- JSON format and
custom-columns
output mode are supported through the--output
flag. - It is possible to filter events by container name using the
--containername
flag.
For instance, for the list-containers
command:
$ sudo local-gadget list-containers -o json --containername etcd
[
{
"runtime": "containerd",
"id": "fef9c7f66e0d68c554b7ea48cc3ef4e77c553957807de7f05ad0210a05d8c215",
"pid": 1611,
"mntns": 4026532270,
"netns": 4026531992,
"cgroupPath": "/sys/fs/cgroup/unified/system.slice/containerd.service",
"cgroupID": 854,
"cgroupV1": "/system.slice/containerd.service/kubepods-burstable-pod87a960e902bbb19289771a77e4b07353.slice:cri-containerd:fef9c7f66e0d68c554b7ea48cc3ef4e77c553957807de7f05ad0210a05d8c215",
"cgroupV2": "/system.slice/containerd.service",
"namespace": "kube-system",
"podname": "etcd-master",
"name": "etcd",
"podUID": "87a960e902bbb19289771a77e4b07353"
}
]
Running some gadgets
Snapshot/Process
$ sudo local-gadget snapshot process
CONTAINER COMM PID
calico-kube-controllers kube-controller 4791
calico-node bird 3888
calico-node bird6 3889
calico-node calico-node 3731
calico-node calico-node 3732
calico-node calico-node 3733
calico-node calico-node 3734
calico-node calico-node 3735
calico-node calico-node 3737
calico-node runsv 3723
calico-node runsv 3724
calico-node runsv 3725
calico-node runsv 3726
calico-node runsv 3727
calico-node runsv 3728
calico-node runsv 3729
calico-node runsv 3730
calico-node runsvdir 3643
coredns coredns 4725
coredns coredns 4762
etcd etcd 1750
gadget gadgettracerman 39645
kube-apiserver kube-apiserver 1793
kube-controller-manager kube-controller 1788
kube-proxy kube-proxy 3227
kube-scheduler kube-scheduler 1840
myContainer nginx 34187
myContainer nginx 34279
myContainer nginx 34280
myContainer nginx 34281
myContainer nginx 34282
We can filter by container name:
$ sudo local-gadget snapshot process --containername gadget
CONTAINER COMM PID
gadget gadgettracerman 39645
And, show the all threads using the -t
flag:
$ sudo local-gadget snapshot process --containername gadget -t
CONTAINER COMM TGID PID
gadget gadgettracerman 39645 39645
gadget gadgettracerman 39645 39668
gadget gadgettracerman 39645 39669
gadget gadgettracerman 39645 39670
gadget gadgettracerman 39645 39671
gadget gadgettracerman 39645 39672
gadget gadgettracerman 39645 39673
gadget gadgettracerman 39645 39674
gadget gadgettracerman 39645 39677
gadget gadgettracerman 39645 39678
Trace/Bind
$ sudo local-gadget trace bind
CONTAINER PID COMM PROTO ADDR PORT OPTS IF
foo 380299 nc TCP :: 4242 .R... 0
The previous output was trigged using the following test container:
$ docker run -it --rm --name foo busybox /bin/sh -c "nc -l -p 4242"
In case of need, we can specify the ports we want to monitor:
$ sudo local-gadget trace bind --ports 4242
Use local-gadget trace bind --help
to discover the rest of the filtering
options available for this gadget.
Trace/Exec
This is the output when executing this gadget on a Kubernetes node:
$ sudo local-gadget trace exec
CONTAINER PID PPID PCOMM RET ARGS
calico-node 416789 416777 calico-node 0 /bin/calico-node -felix-live -bird-live
calico-node 416804 416789 sv 0 /usr/local/bin/sv status /etc/service/enabled/confd
calico-node 416805 416789 sv 0 /usr/local/bin/sv status /etc/service/enabled/bird
gadget 416816 416806 gadgettracerman 0 /bin/gadgettracermanager -liveness
gadget 416842 416823 gadgettracerman 0 /bin/gadgettracermanager -liveness
calico-node 416887 416876 calico-node 0 /bin/calico-node -felix-ready -bird-ready
Remember that we can use the -o custom-columns
flag to show only the columns
we are interested in:
$ sudo local-gadget trace exec -o custom-columns=container,pid,pcomm
CONTAINER PID PCOMM
calico-node 421023 ipset
calico-node 421039 calico-node
calico-node 421056 sv
gadget 421066 gadgettracerman
Trace/Tcp
We can also monitor the TCP connections using the tcp trace gadget. For instance, with the following container we can see that the gadget shows that a TCP connection was established:
$ docker run -it --rm --name test-container busybox /bin/sh -c "wget https://www.example.com"
Connecting to www.kinvolk.io (188.114.96.7:443)
saving to 'index.html'
index.html 100% |index.html 100% |**********************************| 36362 0:00:00 ETA
'index.html' saved
$ sudo local-gadget trace tcp
CONTAINER T PID COMM IP SADDR DADDR SPORT DPORT
test-container C 11039 wget 4 172.17.0.2 188.114.96.7 57560 443
Trace/TcpConnect
The tcpconnect trace gadget traces IPv4 and IPv6 TCP connections.
$ docker run -it --rm --name test-container busybox /bin/sh -c "wget http://www.example.com"
Connecting to www.example.com (93.184.216.34:80)
saving to 'index.html'
index.html 100% |************************************************************************************************| 1256 0:00:00 ETA
'index.html' saved
$ sudo local-gadget trace tcpconnect --containername test-container
CONTAINER PID COMM IP SADDR DADDR DPORT
test-container 503650 wget 4 172.17.0.3 93.184.216.34 80
Trace/Signal
The signal trace gadget is used to trace system signals received by containers.
$ docker run -it --rm --name test-container busybox /bin/sh
/ # sleep 100 &
/ # echo $!
7
/ # kill -kill $!
/ # exit
$ sudo local-gadget trace signal --containername test-container
WARN[0000] Runtime enricher (containerd): couldn't get current containers
WARN[0000] Runtime enricher (cri-o): couldn't get current containers
CONTAINER PID COMM SIGNAL TPID RET
test-container 11131 sh SIGKILL 11162 0
test-container 11131 sh SIGKILL 7 0
test-container 11131 sh SIGHUP 11131 0
Using the interactive mode
The interactive mode allows us to create multiple traces at the same time.
Use the list-gadgets
commands to verify the supported gadgets:
$ sudo local-gadget interactive
» list-gadgets
audit-seccomp
dns
network-graph
process-collector
seccomp
snisnoop
socket-collector
Following are some examples of usage.
dns
Start the DNS gadget:
$ sudo local-gadget interactive --runtimes docker
» create dns trace1 --container-selector shell01
» stream trace1 -f
{"notice":"tracer attached","node":"local","namespace":"default","pod":"shell01"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"www.wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"www.wikipedia.org.","pktType":"OUTGOING"}
{"notice":"tracer detached","node":"local","namespace":"default","pod":"shell01"}
Start a container:
$ docker run -ti --rm --name shell01 busybox wget wikipedia.org
seccomp
$ sudo local-gadget interactive --runtimes docker
» create seccomp trace1 --container-selector shell01 --output-mode Status
Start a container:
$ docker run -ti --rm --name shell01 busybox
Resume from the local-gadget terminal:
» operation trace1 generate
State: Started
{
"defaultAction": "SCMP_ACT_ERRNO",
"architectures": [
"SCMP_ARCH_X86_64",
"SCMP_ARCH_X86",
"SCMP_ARCH_X32"
],
"syscalls": [
{
"names": [
"arch_prctl",
"brk",
"close",
"fcntl",
"getcwd",
"geteuid",
"getpgrp",
"getpid",
"getppid",
"getuid",
"ioctl",
"open",
"poll",
"read",
"rt_sigaction",
"rt_sigreturn",
"setpgid",
"write"
],
"action": "SCMP_ACT_ALLOW"
}
]
}