local-gadget

    Inspektor Gadget can also be used without Kubernetes to trace containers with the local-gadget tool.

    Currently, the local-gadget can trace containers managed by Docker regardless of whether they were created via Kubernetes or not. In addition, it will also use the CRI to trace containers managed by containerd and CRI-O, meaning only the ones created via Kubernetes. Support for non-Kubernetes containers with containerd is coming, see issue #734 .

    By default, the local-gadget will try to communicate with the Docker Engine API and the CRI API of containerd and CRI-O:

    $ docker run -d --name myContainer nginx:1.21
    95b814bb82b9e30dd935b03d04a7b00b6978ce018a6f55d6a9c7a824b31ec6b5
    
    $ sudo local-gadget list-containers
    WARN[0000] Runtime enricher (cri-o): couldn't get current containers
    RUNTIME       ID               NAME
    containerd    7766d32caded4    calico-kube-controllers
    containerd    2e3e4968b456f    calico-node
    containerd    d3be7741b94ff    coredns
    containerd    e7be3e4dc1bb4    coredns
    containerd    fb4fe41921f30    etcd
    containerd    136e7944d2077    kube-apiserver
    containerd    ad8709a2c2ded    kube-controller-manager
    containerd    66cf05654a47f    kube-proxy
    containerd    a68bed42aa6b2    kube-scheduler
    docker        95b814bb82b9e    myContainer
    

    This output shows the containers local-gadget retrieved from Docker and containerd, while the warning message tells us that local-gadget tried to communicate with CRI-O but couldn’t. In this case, it was because CRI-O was not running in the system where we executed the test. However, it could also happen if local-gadget uses a different UNIX socket path to communicate with the runtimes. To check which paths local-gadget is using, you can use the --help flag:

    $ sudo local-gadget list-containers --help
    List all containers
    
    Usage:
      local-gadget list-containers [flags]
    
    Flags:
      ...
          --containerd-socketpath string   containerd CRI Unix socket path (default "/run/containerd/containerd.sock")
          --crio-socketpath string         CRI-O CRI Unix socket path (default "/run/crio/crio.sock")
          --docker-socketpath string       Docker Engine API Unix socket path (default "/run/docker.sock")
      -r, --runtimes string                Container runtimes to be used separated by comma. Supported values are: docker, containerd, cri-o (default "docker,containerd,cri-o")
      ...
    

    If needed, we can also specify the runtimes to be used and their UNIX socket path:

    $ sudo local-gadget list-containers --runtimes docker --docker-socketpath /some/path/docker.sock
    RUNTIME    ID               NAME
    docker     95b814bb82b9e    myContainer
    

    The list-containers command also supports JSON format and filtering:

    $ sudo local-gadget list-containers -o json --containername calico-kube-controllers
    [
      {
        "id": "1fd08f8d9fc300a3c312edc1718d31d424744ebbcf70a3ed84a8dc5402a64dc7",
        "pid": 2974,
        "mntns": 4026532290,
        "netns": 4026531992,
        "cgroupPath": "/sys/fs/cgroup/unified/system.slice/containerd.service",
        "cgroupID": 854,
        "cgroupV1": "/system.slice/containerd.service/kubepods-besteffort-pod3b57919c_9239_4b73_a74c_2bdb9c7c2926.slice:cri-containerd:7e069227538c40f2965b9610018678e93e07fadd04add1aa5d5243337038f336",
        "cgroupV2": "/system.slice/containerd.service",
        "namespace": "default",
        "podname": "kube-proxy",
        "name": "kube-proxy",
        "runtime": "containerd"
      }
    ]
    

    Please consider that some of the Kubernetes metadata: Namespace, Podname, PodUID and Name, could not correspond to the actual values, issue #737 will fix this soon. While, the Kubernetes metadata Labels and OwnerReference, and the OCI runtime data: OciConfig and Bundle, will remain empty because such information cannot be retrieved from the container runtime.

    Examples

    We can execute the local-gadget --help flag to check the supported gadgets, following some examples of usage.

    Common features

    Notice that all the gadgets support the following features even if, for simplicity, they are not demonstrated in each gadget’s examples.

    • JSON format and custom-columns output mode are supported through the --output flag.
    • It is possible to filter events by container name using the --containername flag.

    Snapshot/Process

    $ sudo local-gadget snapshot process
    CONTAINER                  COMM               PID
    calico-kube-controllers    kube-controller    4791
    calico-node                bird               3888
    calico-node                bird6              3889
    calico-node                calico-node        3731
    calico-node                calico-node        3732
    calico-node                calico-node        3733
    calico-node                calico-node        3734
    calico-node                calico-node        3735
    calico-node                calico-node        3737
    calico-node                runsv              3723
    calico-node                runsv              3724
    calico-node                runsv              3725
    calico-node                runsv              3726
    calico-node                runsv              3727
    calico-node                runsv              3728
    calico-node                runsv              3729
    calico-node                runsv              3730
    calico-node                runsvdir           3643
    coredns                    coredns            4725
    coredns                    coredns            4762
    etcd                       etcd               1750
    gadget                     gadgettracerman    39645
    kube-apiserver             kube-apiserver     1793
    kube-controller-manager    kube-controller    1788
    kube-proxy                 kube-proxy         3227
    kube-scheduler             kube-scheduler     1840
    myContainer                nginx              34187
    myContainer                nginx              34279
    myContainer                nginx              34280
    myContainer                nginx              34281
    myContainer                nginx              34282
    

    We can filter by container name:

    $ sudo local-gadget snapshot process --containername gadget
    CONTAINER    COMM               PID
    gadget       gadgettracerman    39645
    

    And, show the all threads using the -t flag:

    $ sudo local-gadget snapshot process --containername gadget -t
    CONTAINER    COMM               TGID     PID
    gadget       gadgettracerman    39645    39645
    gadget       gadgettracerman    39645    39668
    gadget       gadgettracerman    39645    39669
    gadget       gadgettracerman    39645    39670
    gadget       gadgettracerman    39645    39671
    gadget       gadgettracerman    39645    39672
    gadget       gadgettracerman    39645    39673
    gadget       gadgettracerman    39645    39674
    gadget       gadgettracerman    39645    39677
    gadget       gadgettracerman    39645    39678
    

    Trace/Bind

    $ sudo local-gadget trace bind
    CONTAINER        PID     COMM             PROTO  ADDR             PORT    OPTS    IF
    foo              380299  nc               TCP    ::               4242    .R...   0
    

    The previous output was trigged using the following test container:

    $ docker run -it --rm --name foo busybox /bin/sh -c "nc -l -p 4242"
    

    In case of need, we can specify the ports we want to monitor:

    $ sudo local-gadget trace bind --ports 4242
    

    Use local-gadget trace bind --help to discover the rest of the filtering options available for this gadget.

    Trace/Exec

    This is the output when executing this gadget on a Kubernetes node:

    $ sudo local-gadget trace exec
    CONTAINER        PID     PPID    PCOMM            RET  ARGS
    calico-node      416789  416777  calico-node      0    /bin/calico-node -felix-live -bird-live
    calico-node      416804  416789  sv               0    /usr/local/bin/sv status /etc/service/enabled/confd
    calico-node      416805  416789  sv               0    /usr/local/bin/sv status /etc/service/enabled/bird
    gadget           416816  416806  gadgettracerman  0    /bin/gadgettracermanager -liveness
    gadget           416842  416823  gadgettracerman  0    /bin/gadgettracermanager -liveness
    calico-node      416887  416876  calico-node      0    /bin/calico-node -felix-ready -bird-ready
    

    Remember that we can use the -o custom-columns flag to show only the columns we are interested in:

    $ sudo local-gadget trace exec -o custom-columns=container,pid,pcomm
    CONTAINER        PID     PCOMM
    calico-node      421023  ipset
    calico-node      421039  calico-node
    calico-node      421056  sv
    gadget           421066  gadgettracerman
    

    Trace/Tcp

    We can also monitor the TCP connections using the tcp trace gadget. For instance, with the following container we can see that the gadget shows that a TCP connection was established:

    $ docker run -it --rm --name test-container busybox /bin/sh -c "wget https://www.example.com"
    Connecting to www.kinvolk.io (188.114.96.7:443)
    saving to 'index.html'
    index.html           100% |index.html           100% |**********************************| 36362  0:00:00 ETA
    'index.html' saved
    
    $ sudo local-gadget trace tcp
    CONTAINER        T  PID     COMM             IP  SADDR                  DADDR                  SPORT   DPORT
    test-container   C  11039   wget             4   172.17.0.2             188.114.96.7           57560   443
    

    Trace/TcpConnect

    The tcpconnect trace gadget traces IPv4 and IPv6 TCP connections.

    $ docker run -it --rm --name test-container busybox /bin/sh -c "wget http://www.example.com"
    Connecting to www.example.com (93.184.216.34:80)
    saving to 'index.html'
    index.html           100% |************************************************************************************************|  1256  0:00:00 ETA
    'index.html' saved
    
    $ sudo local-gadget trace tcpconnect --containername test-container
    CONTAINER        PID     COMM             IP  SADDR            DADDR            DPORT
    test-container   503650  wget             4   172.17.0.3       93.184.216.34    80
    

    Interactive Mode

    The interactive mode allows us to create multiple traces at the same time.

    Use the list-gadgets commands to verify the supported gadgets:

    $ sudo local-gadget interactive
    » list-gadgets
    audit-seccomp
    dns
    network-graph
    process-collector
    seccomp
    snisnoop
    socket-collector
    

    Following are some examples of usage.

    dns

    Start the DNS gadget:

    $ sudo local-gadget interactive --runtimes docker
    » create dns trace1 --container-selector shell01
    » stream trace1 -f
    {"notice":"tracer attached","node":"local","namespace":"default","pod":"shell01"}
    {"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
    {"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
    {"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
    {"node":"local","namespace":"default","pod":"shell01","name":"wikipedia.org.","pktType":"OUTGOING"}
    {"node":"local","namespace":"default","pod":"shell01","name":"www.wikipedia.org.","pktType":"OUTGOING"}
    {"node":"local","namespace":"default","pod":"shell01","name":"www.wikipedia.org.","pktType":"OUTGOING"}
    {"notice":"tracer detached","node":"local","namespace":"default","pod":"shell01"}
    

    Start a container:

    $ docker run -ti --rm --name shell01 busybox wget wikipedia.org
    

    seccomp

    $ sudo local-gadget interactive --runtimes docker
    » create seccomp trace1 --container-selector shell01 --output-mode Status
    

    Start a container:

    $ docker run -ti --rm --name shell01 busybox
    

    Resume from the local-gadget terminal:

    » operation trace1 generate
    State: Started
    {
      "defaultAction": "SCMP_ACT_ERRNO",
      "architectures": [
        "SCMP_ARCH_X86_64",
        "SCMP_ARCH_X86",
        "SCMP_ARCH_X32"
      ],
      "syscalls": [
        {
          "names": [
            "arch_prctl",
            "brk",
            "close",
            "fcntl",
            "getcwd",
            "geteuid",
            "getpgrp",
            "getpid",
            "getppid",
            "getuid",
            "ioctl",
            "open",
            "poll",
            "read",
            "rt_sigaction",
            "rt_sigreturn",
            "setpgid",
            "write"
          ],
          "action": "SCMP_ACT_ALLOW"
        }
      ]
    }