Exporting Profiles (OpenTelemetry)

warning

The OpenTelemetry profiling signal is experimental and this support could change without any notice.

Inspektor Gadget supports exporting the experimental profile signal to OpenTelemetry using OTLP. Profile exporters are configured on the otel-profiles section of the config file:

# ig-config.yaml
operator:
  otel-profiles:
    exporters:
      my-profiles-exporter:
        exporter: otlp-grpc
        compression: gzip
        endpoint: "127.0.0.1:4317"
        insecure: true

This will configure an exporter named my-profiles-exporter with the given endpoint, gzip compression enabled and TLS disabled.

You can then run a gadget and activate the exporter for it by setting the --otel-profiles-exporter=my-profiles-exporter flag.

Exporter settings

exporter

Currently we only support otlp-grpc.

compression

Compression can be set to either "none" (no compression) or "gzip" (gzip compression).

endpoint

IP address and port of the gRPC receiver.

insecure

If set to true, the gRPC connection will not use TLS encryption. False by default.

Annotations

Datasource

The exporter looks for some annotations in the datasource to identify which fields to use.

profiles.stack-fields

Defines which fields contain the stack trace of the sample.

profiles.value-field

Defines which field contains the value to use for the sample. For instance, it can be the number of times the stack trace has been found, of the quantity of memory allocated by a given stack trace.

Fields

The exporter also uses some annotations on the fields

profiles.sample-attribute

Marks a field to be used as an attribute of the sample.

Sample attribute names are the full field names, following the OpenTelemetry convention of dot-separated namespaces (e.g. k8s.podName, proc.comm). Pyroscope >= 2.0 keeps them as-is, while Pyroscope 1.x replaced dots with underscores. To keep queries and dashboards written for Pyroscope 1.x working with Pyroscope >= 2.0, use the --otel-profiles-sanitize-labels flag: the exporter then replaces dots with underscores, so k8s.namespace, k8s.podName, and proc.comm are exported as k8s_namespace, k8s_podName, and proc_comm. Field names in gadget metadata and annotations still use dots.

With --otel-profiles-sanitize-labels, if two selected fields normalize to the same attribute name (for example, k8s.podName and k8s_podName), the exporter fails to start with an error rather than overwriting an attribute. Resource attributes such as service.name and service.version retain their OpenTelemetry names.

Previously stored profiles are not renamed: update queries when toggling the flag or upgrading Pyroscope.

Annotations example:

datasources:
  samples:
    annotations:
      profiles.stack-fields: kern_stack
      profiles.value-field: samples
    fields:
      runtime.containerName:
        annotations:
          profiles.sample-attribute: "true"
      proc.comm:
        annotations:
          profiles.sample-attribute: "true"

This configuration indicates that kern_stack field contains the stack trace, the samples field contains the stack trace value and the runtime.containerName and proc.comm fields are used as attributes of the collected samples.

Guide

This guide provides a working end to end example of this support using the profile_cpu gadget, pyroscope and Grafana.

Create the following files on your machine:

# docker-compose.yaml
services:
  pyroscope:
    # fixed versions for pyroscope, otel-collector, otel-profiler due to protocol changes
    image: grafana/pyroscope:1.15.0
    command: ["-self-profiling.disable-push=true"]
    ports:
      - "4040:4040"
    networks:
      - otel-net

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_INSTALL_PLUGINS=grafana-pyroscope-app
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning
    ports:
      - "3000:3000"
    networks:
      - otel-net

networks:
  otel-net:
    driver: bridge

# grafana-provisioning/datasources/pyroscope.yml
---
apiVersion: 1
datasources:
  - uid: local-pyroscope
    type: grafana-pyroscope-datasource
    name: Pyroscope
    url: http://pyroscope:4040
    jsonData:
      keepCookies: [pyroscope_git_session]
    # Uncomment these if using with Grafana Cloud
    # basicAuth: true
    # basicAuthUser: '123456'
    # secureJsonData:
      # basicAuthPassword: PASSWORD

Start the grafana and pyroscope containers using docker compose:

$ docker compose up
...

The Grafana dashboard should be accessible on http://localhost:3000.

Now, configure a profiles exporter pointing to the running pyroscope instance. (Note: We could also configure an instance of the opentelemetry-collector between IG and pyroscope, but for simplicity we avoid this.)

# ig-config.yaml
operator:
  otel-profiles:
    exporters:
      my-profiles-exporter:
        exporter: otlp-grpc
        compression: gzip
        endpoint: "127.0.0.1:4040"
        insecure: true

Then, run the profile_cpu gadget configured to export profiles:

$ sudo ig run profile_cpu --otel-profiles-exporter=my-profiles-exporter --config=ig-config.yaml

Run a container to generate some events:

$ docker run --rm -it --name mycontainer busybox sh -c "cat /dev/random > /dev/null"

The profiles will be available under "Drilldown -> Profiles"

Profiles on Grafana

It's possible to filter data by proc_comm and runtime_containerName:

Profiles filtering