Common Features

Inspektor Gadget comes with a wide array of gadgets that allow us to inspect what’s going on in our clusters. The flags and parameters that we can pass to each specific gadget may differ, but there are some that are supported across all gadgets.

Event Filtering

There are several ways to choose the pods or containers that we want to trace:

--node string, show only data from pods running in that node
-n string, --namespace string, show data from pods in that namespace
-A, --all-namespaces, show data from pods in all namespaces
-p string, --podname string, show only data from pods with that name
-c string, --containername string, show only data from containers with that name
-l string, --selector string: show only data that matches the given label or selector. Only = is currently supported (e.g. key1=value1,key2=value2).

We can use one or more of these parameters to choose which pods or containers will be inspected by our gadgets.

For example:

$ kubectl gadget trace exec -n demo -l app=myapp

Will run the exec tracer for all pods in the demo namespace that have the app=myapp label.

$ kubectl gadget snapshot socket -A -p nginx

Will get the socket snapshot for all pods with name nginx, regardless of which namespace they are in.

Output Format

The -o or --output flag lets us decide the format for the output the gadget will generate. The default columns output shows some of the information gathered, arranged in text columns on the console.

This can be overridden with:

json
jsonpretty
yaml
columns

JSON Output

Passing -o json will print all the information gathered in JSON format. Each entry is printed on a single line, so the output can be easily parsed line by line.

For example:

$ kubectl gadget trace tcp -A -o json | jq
{
  "type": "normal",
  "node": "minikube",
  "namespace": "kube-system",
  "pod": "coredns-66bff467f8-8ftjm",
  "container": "coredns",
  "pid": 2688,
  "comm": "coredns",
  "ipversion": 4,
  "saddr": "127.0.0.1",
  "daddr": "127.0.0.1",
  "sport": 46010,
  "dport": 8080,
  "operation": "connect"
}

JSON Pretty Output

Passing -o jsonpretty will print all the information gathered in JSON format but with indentation making it easier to read.

YAML Output

Passing -o yaml will print all the information gathered in YAML format. Each entry is preceded by the end of directives markers (---).

Custom Columns

Using -o columns=column1,column2 we can choose which columns to print. You can use gadget help to see which columns are available:

$ kubectl gadget trace open -h
...
  Available columns:
    comm
    err
    fd
    flags
    fullPath (requires --full-path)
    gid
    k8s.container
...

For example, when tracing which processes were killed because of the node running out of memory, we can choose to only print the PID and command of the killed process:

$ kubectl gadget trace oomkill -A -o columns=kpid,kcomm
KPID   KCOMM
15182  tail

Also, we can use the + and - prefixes to add or remove columns relative to the default columns. For example, in case we want to add uid and gid columns to the default columns for trace open gadget:

$ kubectl gadget trace open -A -o columns=+uid,+gid
K8S.NODE      K8S.NAMESPACE K8S.POD       K8S.CONTAINER PID     COMM   FD ERR PATH                     UID        GID
miniku…docker default       test-p…-v6rqg nginx         1149213 docke… 3  0   /etc/ld.so.cache         0          0

Columns k8s and runtime are expanded to all available columns for the respective environment. For example, in case we want trace tcp gadget to print events with only runtime columns:

$ kubectl gadget trace tcp -o columns=-k8s,+runtime
RUNTIME.RUNTIMENAME RUNTIM… RUNTIME.CONTAIN… RUNTIME… T PID      COMM     IP SRC                    DST
docker              8df2cb… k8s_nginx_test-… nginx    A 1163696  nginx    4  p/default/test-pod-67c r/10.244.0.1:58570
docker              8df2cb… k8s_nginx_test-… nginx    X 1163696  nginx    4  p/default/test-pod-67c r/10.244.0.1:58570

Run for a specific amount of time

Many gadgets will run forever, printing the gathered output until we press Ctrl-C to stop them. If we want to run a gadget only for a window of time, we can use the --timeout int flag, passing the number of seconds during which we want to run the gadget.

For example, we can trace files that get opened by pods in the gadget namespace during a window of 5 seconds, like this:

$ kubectl gadget trace open -n gadget --timeout 5
K8S.NODE         K8S.NAMESPACE    K8S.POD          K8S.CONTAINER    PID     COMM             FD    ERR PATH
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  3     0   /etc/ld.so.cache
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  3     0   /lib/x86_64-linux-gnu/libpthread.so.0
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  3     0   /lib/x86_64-linux-gnu/libseccomp.so.2
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  3     0   /lib/x86_64-linux-gnu/libc.so.6
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  3     0   /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  6     0   /usr/bin/gadgettracermanager
minikube         gadget           gadget-vhcj7     gadget           1303299 gadgettracerman  6     0   /etc/localtime

Kubernetes CLI Runtime options

The Inspektor Gadget kubectl plugin uses the kubernetes cli-runtime helpers. This adds support for many CLI options that are common to many Kubernetes tools, which let us specify how to connect to the cluster, which kubeconfig to use, and so on.

  --as string                      Username to impersonate for the operation
  --as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
  --cache-dir string               Default cache directory (default "/home/marga/.kube/cache")
  --certificate-authority string   Path to a cert file for the certificate authority
  --client-certificate string      Path to a client certificate file for TLS
  --client-key string              Path to a client key file for TLS
  --cluster string                 The name of the kubeconfig cluster to use
  --context string                 The name of the kubeconfig context to use
  --insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
  --kubeconfig string              Path to the kubeconfig file to use for CLI requests.
  --request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
, --server string                  The address and port of the Kubernetes API server
  --tls-server-name string         Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
  --token string                   Bearer token for authentication to the API server
  --user string                    The name of the kubeconfig user to use

If none of these options are specified, Inspektor Gadget will connect to the cluster configured in the default kubeconfig location, with the default connection options.