Skip to main content
Version: v0.55.0

profile_cpu

The profile cpu gadget takes samples of stack traces.

Requirements​

  • Minimum Kernel Version : *5.4

*This is the minimal kernel version we have tried for this Gadget, however it's possible that it works with earlier versions.

Getting started​

Running the gadget:

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:v0.55.0 --map-fetch-interval 0 [flags]

Guide​

Here we deploy a small demo pod "random":

$ kubectl run --restart=Never --image=busybox random -- sh -c 'cat /dev/urandom > /dev/null'
pod/random created

Using the profile cpu gadget, we can see the list of stack traces. The following command filters only for pods named "random", execute the command and interrupt it after ~30 seconds. The --kernel-stacks-only option is passed to show only the kernel stack traces.

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:v0.55.0 --podname random --kernel-stacks-only --map-fetch-interval 0

After a while press with Ctrl-C to stop trace collection

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:v0.55.0 --timeout 5 --podname random --kernel-stacks-only
K8S.NODE                 K8S.NAMESPACE            K8S.PODNAME             K8S.CONTAINERNAME              PID COMM             SAMPLES              KERN_STACK
minikube-docker          default                  random                  random                       38130 containerd-shim  5                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_block_generic;…
minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_permute; [1]ge…

From the traces above, you can see that the pod is spending CPU time in the Linux function urandom_read.

Instead of waiting, you can use the --timeout argument:

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_cpu:v0.55.0 --timeout 5 --podname random --kernel-stacks-only --map-fetch-interval 0
K8S.NODE                 K8S.NAMESPACE            K8S.PODNAME             K8S.CONTAINERNAME              PID COMM             SAMPLES              KERN_STACK
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]_copy_to_iter; [1]get…
minikube-docker          default                  random                  random                       38130 containerd-shim  2                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  5                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]chacha_permute; [1]ge…
minikube-docker          default                  random                  random                       38130 containerd-shim  1                    [0]get_random_bytes_user…

Finally, we need to clean up our pod:

$ kubectl delete pod random

GPU-idle attribution (--gpu-idle-only)​

On a machine running the gpu-ebpf-bridge, the --gpu-idle-only flag turns profile_cpu into a GPU-starvation profiler: it only records a stack sample when the sampled process owns GPU memory and its GPU is underutilized at that instant. The resulting flamegraph shows exactly the CPU code that runs while the GPU the process paid for sits idle — the on-CPU work sitting between successive GPU kernels.

Because it is sampling-based, it has no dependence on the bridge poll interval and therefore also catches "many small gaps" starvation (for example a per-frame CPU preprocessing step that keeps an inference GPU at only ~50% duty cycle), which the recency-based trace_gpu_starvation gadget cannot see.

$ sudo ig run ghcr.io/inspektor-gadget/gadget/profile_cpu:v0.55.0 \
    --gpu-idle-only

Relevant flags:

  • --gpu-idle-only: enable the filter (default false).
  • --gpu-util-max-pct: only record a sample when the primary GPU's SM utilization is below this percentage (default 70).
  • --min-gpu-mem-bytes: minimum resident GPU memory for a process to count as a GPU holder (default 1, i.e. any allocation).
  • --stale-threshold-ms: discard samples when the bridge data is older than this (default 300). If the bridge is not running the maps are empty and every sample is discarded, yielding an empty profile rather than mis-attributed stacks.

Note on symbolization: by default --gpu-idle-only symbolizes stacks from native symbol tables only (the user_stack_raw.symbols field), so C/C++ and other compiled frames (for example cv::resize / numpy) resolve. To also get interpreted-language frames — most importantly Python — add --collect-otel-stack --symbolizers otel-ebpf-profiler, which unwinds the interpreter stack via the OTel eBPF profiler (supported for profile_cpu's perf_event program type).