Skip to main content
Version: main

Verifying Gadgets

Inspektor Gadget official gadgets are signed using cosign. In this guide, we will see how you can verify them with this tool as well as verifying your own gadgets using one of the supported methods, i.e. cosign and notation.

Verify with cosign​

Like our container image, we sign all our image-based gadgets. The signatures are by default verified against the official Inspektor Gadget public key.

As of now, we support the following cosign format:

When using Inspektor Gadget on Kubernetes, you have to configure these features at deploy time. This enforces the policies so that users cannot override them. By default all image-based gadgets are verified using Inspektor Gadget public key:

$ kubectl gadget deploy
...
Inspektor Gadget successfully deployed
$ kubectl gadget run ghcr.io/your_repo/trace_exec -A
Error: fetching gadget information: getting gadget info: rpc error: code = Unknown desc = getting gadget info: initializing and preparing operators: instantiating operator "oci": ensuring image: verifying image "ghcr.io/your_repo/trace_exec": verifying signature: invalid signature when validating ASN.1 encoded signature
$ kubectl gadget run trace_exec -A
K8S.NAMESP… K8S.PODNAME K8S.CONTAI… MNTNS… TIMES… PID    PPID   UID    GID    LOGIN… SESSI… RETVAL ARGS_… UPPER… ARGS_… COMM  ARGS  K8S.…
gadget      gadge…hbh8n gadget      40265… 22867… 53386  53368  0      0      42949… 42949… 0      2      false  35     gadg… /bin… mini…
gadget      gadge…hbh8n gadget      40265… 22867… 53387  53369  0      0      42949… 42949… 0      2      false  35     gadg… /bin… mini…
...

You can specify a custom public key at deployment time using --daemon-config. Start by creating daemon config file:

cat <<EOF > daemon-config.yaml
operator:
    oci:
        public-keys:
        - |
            -----BEGIN PUBLIC KEY-----
            MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp3pbyFZOnEz4ZtkXY0Zm
            Q78zbdZ9Pyhgje+oD1T49r7HYeUdf2vKyc1ObcCwT3EZyzv48IC6cBcdNu4RjbS8
            snpzFVK5P9szolYJfuMJyLkMTGOAc6AjA0YY8b3wJtV8F3N3IjzA0HtJ3vIqDxIb
            e8rm52uh4RV7SooxMyh5SAdPi06/kAV2ZClQK1z5Vob9zM2U8w2M8Ib4h/cHBY4e
            7W9IuQZTvwlAEyEZng7GUPf/dke69/9I3/N9UoKXIJ8cHg0pxtWKPmCGoFqFT9qR
            ehd9nJdWmLfFT7y1KmfkxknC0TAl3lA4ArZqYYzRmX8cUFz2mGNzNnWgkxN7qVMS
            LQIDAQAB
            -----END PUBLIC KEY-----
EOF

Then, you can deploy Inspektor Gadget with this configuration:


$ kubectl gadget deploy --daemon-config=daemon-config.yaml
...
Inspektor Gadget successfully deployed
# Without the official public key, official gadgets will be rejected.
$ kubectl gadget run trace_exec -A
Error: fetching gadget information: getting gadget info: rpc error: code = Unknown desc = getting gadget info: initializing and preparing operators: instantiating operator "oci": ensuring image: verifying image "trace_exec": the image was not signed by the provided keys: crypto/rsa: verification error
$ kubectl-gadget run ghcr.io/your_repo/trace_exec -A
K8S.NAMESP… K8S.PODNAME K8S.CONTAI… MNTNS… TIMES… PID    PPID   UID    GID    LOGIN… SESSI… RETVAL ARGS_… UPPER… ARGS_… COMM  ARGS  K8S.…
gadget      gadge…hbh8n gadget      40265… 22867… 53386  53368  0      0      42949… 42949… 0      2      false  35     gadg… /bin… mini…
gadget      gadge…hbh8n gadget      40265… 22867… 53387  53369  0      0      42949… 42949… 0      2      false  35     gadg… /bin… mini…
...

Verify with notation​

Along with cosign, Inspektor Gadget supports running gadgets signed with notation. The main difference consists in using certificates and a policy document. Note that, notation and cosign verification are complementary and it is possible to run gadgets signed with both methods.

When using Inspektor Gadget on Kubernetes, you have to configure these features at deploy time. This enforces the policies so that users cannot override them.

You can specify a certificate at deployment time using --daemon-config. Start by creating daemon config file:

cat <<EOF > daemon-config.yaml
operator:
    oci:
        notation-certificates:
        - |
            -----BEGIN CERTIFICATE-----
            MIIDRTCCAi2gAwIBAgICAIEwDQYJKoZIhvcNAQELBQAwUTELMAkGA1UEBhMCVVMx
            CzAJBgNVBAgTAldBMRAwDgYDVQQHEwdTZWF0dGxlMQ8wDQYDVQQKEwZOb3Rhcnkx
            EjAQBgNVBAMTCWJhcmZvby5pbzAeFw0yNTEwMDExNTQ2MzVaFw0yNTEwMDIxNTQ2
            MzVaMFExCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJXQTEQMA4GA1UEBxMHU2VhdHRs
            ZTEPMA0GA1UEChMGTm90YXJ5MRIwEAYDVQQDEwliYXJmb28uaW8wggEiMA0GCSqG
            SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDirG5+wdavupHd+K3M4hvPAzQu1NE6Edur
            VsptZTBXnZNCT3/rVlzIF6uoExUe9k8xdBV86aVlI1KUKXb3jWtq4u+9ACcmZ3nO
            +JzhkOZEGN7XTf7axDWIZWv5mye655shIwtLrrguNBs+0lXqO+x5uImqdGtjgH5S
            RrDpHzSNiRziyqym1fCgJI3lwO/sWIZK+2oJLPvCYzS4k0sNo50wB6JI+Eu/qtBK
            c0OCXLhKr0ml9bRpDaoMyIhQrD0+k5JWSDPlNCwIEYueJpf3Ua38YuuM3pPEj8MA
            /hzEyrFpqeJV2Nn5JHRjAecSZbFRHyRtzETn9NKaz00EnoE3/O3nAgMBAAGjJzAl
            MA4GA1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAzANBgkqhkiG9w0B
            AQsFAAOCAQEANjfgsg6Zxc/g+8T3ZLNMj0myd8yciiL16iS4+LQKViRGNmAjRbic
            eslrAFXSiZHI/RIJxKnQBxtDiXs8UlknV2mLnxZP+S2vk3kW/z0w6gRPRC8t3w3K
            BcsDhE//PzLmnXdEYSZQvgXfdv2JujEb1euspyTjRL1vBlxdPh5eeevZsysHrt6q
            c2bFCX/8nicguibCC4ybKQazsbQK07tIPDwlbu0+sCk6ojmhdMZeOaXVcUoaisgq
            dKl8IemhdshLNii2f05fLb1ftcMaG0XIT7i86txiDbiBs+srcjW4bqfhSc201jHu
            /ftuKYyrrWBgnWZ6gmzqC5SR4pBdZVAp3w==
            -----END CERTIFICATE-----
        notation-policy-document: |
          {
              "version": "1.0",
              "trustPolicies": [
                  {
                      "name": "inspektor-gadget-policy",
                      "registryScopes": [ "*" ],
                      "signatureVerification": {
                        "level" : "strict"
                      },
                      "trustStores": ["ca:foobar.io"],
                      "trustedIdentities": [
                        "x509.subject: CN=foobar.io, O=Notary, L=Seattle, ST=WA, C=US"
                      ]
                  }
              ]
          }
EOF

Then, you can deploy Inspektor Gadget with this configuration:


$ kubectl gadget deploy --daemon-config=daemon-config.yaml
...
Inspektor Gadget successfully deployed
$ kubectl-gadget run ghcr.io/your_repo/trace_exec -A
K8S.NAMESP… K8S.PODNAME K8S.CONTAI… MNTNS… TIMES… PID    PPID   UID    GID    LOGIN… SESSI… RETVAL ARGS_… UPPER… ARGS_… COMM  ARGS  K8S.…
gadget      gadge…hbh8n gadget      40265… 22867… 53386  53368  0      0      42949… 42949… 0      2      false  35     gadg… /bin… mini…
gadget      gadge…hbh8n gadget      40265… 22867… 53387  53369  0      0      42949… 42949… 0      2      false  35     gadg… /bin… mini…
...

Disabling the verification​

You can skip verifying image-based gadget signature. Note that we do not recommend using this and this should be used for development purpose only:

Start by creating a daemon config file:

cat <<EOF > daemon-config.yaml
operator:
    oci:
        verify-image: false
EOF

Then, you can deploy Inspektor Gadget with this configuration:

$ kubectl gadget deploy --daemon-config=daemon-config.yaml
...
Inspektor Gadget successfully deployed
$ kubectl gadget run trace_exec -A
WARN[0001] minikube-docker      | gadget signature verification is disabled due to using corresponding option
K8S.NAMESPACE       K8S.PODNAME         K8S.CONTAINERNAME          PID       PPID RE… COMM      ARGS      K8S.NODE  TIMESTAMP
gadget              gadget-z55jq        gadget                   55376      55357   0 gadgettr… /bin/gad… minikube… 2024-07-17T07:39:07.…
gadget              gadget-z55jq        gadget                   55375      55358   0 gadgettr… /bin/gad… minikube… 2024-07-17T07:39:07.…
...