Custom Trace Resource

⚠️ We are currently removing the Trace resource and using a gRPC API instead. Once done, the Trace resource will be deprecated.

Inspektor Gadget uses a custom Trace resource to communicate between the endpoints and the gadget DaemonSet running on the cluster. These resources are used to control which gadgets run in a given cluster.

This is the basic format of this resource:

apiVersion: gadget.kinvolk.io/v1alpha1
kind: Trace
metadata:
  name: trace-name
  namespace: gadget
spec:
  node: node-name
  gadget: gadget-name
  filter:
    namespace: namespace-name
    podname: podname-name
  runMode: Manual
  outputMode: Status

Some gadgets work at the node level, while others support specific filters, like namespace, podname, labels, and so on.

The possible values for outputMode also depend on the gadget. The seccomp gadget, for example, can create seccomp policies as an external resource when ExternalResource is selected. If outputMode is set to Status, the output of the trace will be stored in the status field of the trace resource.

See the corresponding gadgets specs to find out what’s available.

Note that all traces should be created in the gadget namespace. And, for now, the node name needs to be explicitly set in the trace.

Setting the Trace operation

Once the Trace resource is created, the gadget.kinvolk.io/operation field is used to control when the gadget should run.

Some gadgets accept a single operation that triggers a run and stores the output. Other gadgets need to be started and stay running until they get stopped later on to obtain the output.

To control whether a gadget should get started or stopped, annotate the corresponding trace resource with gadget.kinvolk.io/operation=<operation-name>

Note that the trace controller will remove the annotation after it has processed it. So, if you find yourself having to forcefully overwrite the value of this field, it means that the trace controller is having trouble processing your Trace resource.

Using Trace resources from the command line

It’s possible to create and interact with the Trace resources directly from the command line, using kubectl apply with a YAML file containing the specified trace.

After creating the resource, the corresponding operation can be set with kubectl annotate, as in the following example:

$ kubectl annotate -n gadget trace/trace-name gadget.kinvolk.io/operation=start

Using Trace resources from graphical interfaces

Graphical interfaces that interact with Kubernetes, can integrate with Inspektor Gadget by creating and modifying Trace resources in the gadget namespace, and following the corresponding gadget specifications.

Gadgets can then be controlled by annotating the gadget.kinvolk.io/operation field with the corresponding operation.

As an example, see the biolatency plugin implementation for Headlamp.

Interacting with traces using the kubectl-gadget CLI

The kubectl-gadget plugin may create, annotate and delete Trace resources as necessary to interact with the gadget DaemonSet running on the nodes. This is mostly transparent to the user, who will just get the results through the command-line.