local-gadget

Inspektor Gadget relies on the Kubernetes API server to work. However, there are some cases where it is necessary, or preferred, to trace containers without passing through Kubernetes. In such scenarios, you can use the local-gadget tool, as it allows you to collect insights from the nodes to debug your Kubernetes containers without relying on Kubernetes itself, but on the container runtimes. It is important to remark that local-gadget can also be used to trace containers that were not created via Kubernetes.

Some characteristics of the local-gadget:

It uses eBPF as its underlying core technology.
Enriches the collected data with the Kubernetes metadata.
Easy to install as it is a single binary (statically linked).

The architecture of local-gadget is described in the main architecture document.

Use cases

In a Kubernetes environment, when the Kubernetes API server is not working properly, we cannot deploy Inspektor Gadget. Therefore, we still need a way to debug the containers of the cluster.
In some cases, you might have root SSH access to the Kubernetes nodes of a cluster, but not to the kubeconfig.
If you are implementing an application that needs to get insights from the Kubernetes node, you could include the local-gadget binary in your container image, and your app simply execs it. In such a case, it is suggested to use the JSON output format to ease the parsing.
Outside a Kubernetes environment, for observing and debugging standalone containers.

Installation

The instruction to install local-gadget are available in the main installation guide.

Usage

Currently, the local-gadget can trace containers managed by Docker regardless of whether they were created via Kubernetes or not. In addition, it can also use the CRI to trace containers managed by containerd and CRI-O, meaning only the ones created via Kubernetes. Support for non-Kubernetes containers with containerd is coming, see issue #734 .

By default, the local-gadget will try to communicate with the Docker Engine API and the CRI API of containerd and CRI-O:

$ docker run -d --name myContainer nginx:1.21
95b814bb82b9e30dd935b03d04a7b00b6978ce018a6f55d6a9c7a824b31ec6b5

$ sudo local-gadget list-containers
WARN[0000] Runtime enricher (cri-o): couldn't get current containers
RUNTIME       ID               NAME
containerd    7766d32caded4    calico-kube-controllers
containerd    2e3e4968b456f    calico-node
containerd    d3be7741b94ff    coredns
containerd    e7be3e4dc1bb4    coredns
containerd    fb4fe41921f30    etcd
containerd    136e7944d2077    kube-apiserver
containerd    ad8709a2c2ded    kube-controller-manager
containerd    66cf05654a47f    kube-proxy
containerd    a68bed42aa6b2    kube-scheduler
docker        95b814bb82b9e    myContainer

This output shows the containers local-gadget retrieved from Docker and containerd, while the warning message tells us that local-gadget tried to communicate with CRI-O but couldn’t. In this case, it was because CRI-O was not running in the system where we executed the test. However, it could also happen if local-gadget uses a different UNIX socket path to communicate with the runtimes. To check which paths local-gadget is using, you can use the --help flag:

$ sudo local-gadget list-containers --help
List all containers

Usage:
  local-gadget list-containers [flags]

Flags:
  ...
      --containerd-socketpath string   containerd CRI Unix socket path (default "/run/containerd/containerd.sock")
      --crio-socketpath string         CRI-O CRI Unix socket path (default "/run/crio/crio.sock")
      --docker-socketpath string       Docker Engine API Unix socket path (default "/run/docker.sock")
  -r, --runtimes string                Container runtimes to be used separated by comma. Supported values are: docker, containerd, cri-o (default "docker,containerd,cri-o")
  ...

If needed, we can also specify the runtimes to be used and their UNIX socket path:

$ sudo local-gadget list-containers --runtimes docker --docker-socketpath /some/path/docker.sock
RUNTIME    ID               NAME
docker     95b814bb82b9e    myContainer

Common features

Notice that most of the commands support the following features even if, for simplicity, they are not demonstrated in each command guide:

JSON format and custom-columns output mode are supported through the --output flag.
It is possible to filter events by container name using the --containername flag.

For instance, for the list-containers command:

$ sudo local-gadget list-containers -o json --containername etcd
[
  {
    "runtime": "containerd",
    "id": "fef9c7f66e0d68c554b7ea48cc3ef4e77c553957807de7f05ad0210a05d8c215",
    "pid": 1611,
    "mntns": 4026532270,
    "netns": 4026531992,
    "cgroupPath": "/sys/fs/cgroup/unified/system.slice/containerd.service",
    "cgroupID": 854,
    "cgroupV1": "/system.slice/containerd.service/kubepods-burstable-pod87a960e902bbb19289771a77e4b07353.slice:cri-containerd:fef9c7f66e0d68c554b7ea48cc3ef4e77c553957807de7f05ad0210a05d8c215",
    "cgroupV2": "/system.slice/containerd.service",
    "namespace": "kube-system",
    "podname": "etcd-master",
    "name": "etcd",
    "podUID": "87a960e902bbb19289771a77e4b07353"
  }
]

Running some gadgets

Snapshot/Process

$ sudo local-gadget snapshot process
CONTAINER                  COMM               PID
calico-kube-controllers    kube-controller    4791
calico-node                bird               3888
calico-node                bird6              3889
calico-node                calico-node        3731
calico-node                calico-node        3732
calico-node                calico-node        3733
calico-node                calico-node        3734
calico-node                calico-node        3735
calico-node                calico-node        3737
calico-node                runsv              3723
calico-node                runsv              3724
calico-node                runsv              3725
calico-node                runsv              3726
calico-node                runsv              3727
calico-node                runsv              3728
calico-node                runsv              3729
calico-node                runsv              3730
calico-node                runsvdir           3643
coredns                    coredns            4725
coredns                    coredns            4762
etcd                       etcd               1750
gadget                     gadgettracerman    39645
kube-apiserver             kube-apiserver     1793
kube-controller-manager    kube-controller    1788
kube-proxy                 kube-proxy         3227
kube-scheduler             kube-scheduler     1840
myContainer                nginx              34187
myContainer                nginx              34279
myContainer                nginx              34280
myContainer                nginx              34281
myContainer                nginx              34282

We can filter by container name:

$ sudo local-gadget snapshot process --containername gadget
CONTAINER    COMM               PID
gadget       gadgettracerman    39645

And, show the all threads using the -t flag:

$ sudo local-gadget snapshot process --containername gadget -t
CONTAINER    COMM               TGID     PID
gadget       gadgettracerman    39645    39645
gadget       gadgettracerman    39645    39668
gadget       gadgettracerman    39645    39669
gadget       gadgettracerman    39645    39670
gadget       gadgettracerman    39645    39671
gadget       gadgettracerman    39645    39672
gadget       gadgettracerman    39645    39673
gadget       gadgettracerman    39645    39674
gadget       gadgettracerman    39645    39677
gadget       gadgettracerman    39645    39678

Trace/Bind

$ sudo local-gadget trace bind
CONTAINER        PID     COMM             PROTO  ADDR             PORT    OPTS    IF
foo              380299  nc               TCP    ::               4242    .R...   0

The previous output was trigged using the following test container:

$ docker run -it --rm --name foo busybox /bin/sh -c "nc -l -p 4242"

In case of need, we can specify the ports we want to monitor:

$ sudo local-gadget trace bind --ports 4242

Use local-gadget trace bind --help to discover the rest of the filtering options available for this gadget.

Trace/Exec

This is the output when executing this gadget on a Kubernetes node:

$ sudo local-gadget trace exec
CONTAINER        PID     PPID    PCOMM            RET  ARGS
calico-node      416789  416777  calico-node      0    /bin/calico-node -felix-live -bird-live
calico-node      416804  416789  sv               0    /usr/local/bin/sv status /etc/service/enabled/confd
calico-node      416805  416789  sv               0    /usr/local/bin/sv status /etc/service/enabled/bird
gadget           416816  416806  gadgettracerman  0    /bin/gadgettracermanager -liveness
gadget           416842  416823  gadgettracerman  0    /bin/gadgettracermanager -liveness
calico-node      416887  416876  calico-node      0    /bin/calico-node -felix-ready -bird-ready

Remember that we can use the -o custom-columns flag to show only the columns we are interested in:

$ sudo local-gadget trace exec -o custom-columns=container,pid,pcomm
CONTAINER        PID     PCOMM
calico-node      421023  ipset
calico-node      421039  calico-node
calico-node      421056  sv
gadget           421066  gadgettracerman

Trace/Tcp

We can also monitor the TCP connections using the tcp trace gadget. For instance, with the following container we can see that the gadget shows that a TCP connection was established:

$ docker run -it --rm --name test-container busybox /bin/sh -c "wget https://www.example.com"
Connecting to www.kinvolk.io (188.114.96.7:443)
saving to 'index.html'
index.html           100% |index.html           100% |**********************************| 36362  0:00:00 ETA
'index.html' saved

$ sudo local-gadget trace tcp
CONTAINER        T  PID     COMM             IP  SADDR                  DADDR                  SPORT   DPORT
test-container   C  11039   wget             4   172.17.0.2             188.114.96.7           57560   443

Trace/TcpConnect

The tcpconnect trace gadget traces IPv4 and IPv6 TCP connections.

$ docker run -it --rm --name test-container busybox /bin/sh -c "wget http://www.example.com"
Connecting to www.example.com (93.184.216.34:80)
saving to 'index.html'
index.html           100% |************************************************************************************************|  1256  0:00:00 ETA
'index.html' saved

$ sudo local-gadget trace tcpconnect --containername test-container
CONTAINER        PID     COMM             IP  SADDR            DADDR            DPORT
test-container   503650  wget             4   172.17.0.3       93.184.216.34    80

Trace/Signal

The signal trace gadget is used to trace system signals received by containers.

$ docker run -it --rm --name test-container busybox /bin/sh
/ # sleep 100 &
/ # echo $!
7
/ # kill -kill $!
/ # exit

$ sudo local-gadget trace signal --containername test-container
WARN[0000] Runtime enricher (containerd): couldn't get current containers
WARN[0000] Runtime enricher (cri-o): couldn't get current containers
CONTAINER                  PID        COMM          SIGNAL      TPID       RET
test-container             11131      sh            SIGKILL     11162      0
test-container             11131      sh            SIGKILL     7          0
test-container             11131      sh            SIGHUP      11131      0

Using the interactive mode

The interactive mode allows us to create multiple traces at the same time.

Use the list-gadgets commands to verify the supported gadgets:

$ sudo local-gadget interactive
» list-gadgets
audit-seccomp
dns
network-graph
process-collector
seccomp
snisnoop
socket-collector

Following are some examples of usage.

dns

Start the DNS gadget:

$ sudo local-gadget interactive --runtimes docker
» create dns trace1 --container-selector shell01
» stream trace1 -f
{"notice":"tracer attached","node":"local","namespace":"default","pod":"shell01"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"www.wikipedia.org.","pktType":"OUTGOING"}
{"node":"local","namespace":"default","pod":"shell01","name":"www.wikipedia.org.","pktType":"OUTGOING"}
{"notice":"tracer detached","node":"local","namespace":"default","pod":"shell01"}

Start a container:

$ docker run -ti --rm --name shell01 busybox wget wikipedia.org

seccomp

$ sudo local-gadget interactive --runtimes docker
» create seccomp trace1 --container-selector shell01 --output-mode Status

Start a container:

$ docker run -ti --rm --name shell01 busybox

Resume from the local-gadget terminal:

» operation trace1 generate
State: Started
{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": [
    "SCMP_ARCH_X86_64",
    "SCMP_ARCH_X86",
    "SCMP_ARCH_X32"
  ],
  "syscalls": [
    {
      "names": [
        "arch_prctl",
        "brk",
        "close",
        "fcntl",
        "getcwd",
        "geteuid",
        "getpgrp",
        "getpid",
        "getppid",
        "getuid",
        "ioctl",
        "open",
        "poll",
        "read",
        "rt_sigaction",
        "rt_sigreturn",
        "setpgid",
        "write"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}