Contributing
Here you can learn how you can contribute to Inspektor Gadget.
Getting started
To better understand how the pieces fit together, we recommend reading the architecture documentation before starting to play with Inspektor Gadget.
Setup developer environment
-
Fork
and clone this repo:
git clone [email protected]:your_account/inspektor-gadget.git.
- Install Docker , Docker Buildx and Golang .
- To be able to cross build our different container images, you will also need
qemu-user-static.
Building the code
Inspektor Gadget is composed of a client executable that runs on the operator’s machine, and a container image that runs in the Kubernetes cluster. They can be built together or independently.
Building the client executable
You can compile the client executable for your platform by running make kubectl-gadget.
To cross compile for all supported platforms, you can run make kubectl-gadget-all or select a specific one with make kubectl-gadget-linux-amd64 or make kubectl-gadget-darwin-amd64.
Building the gadget container image
You can build and push the container gadget image by running the following commands:
$ make gadget-container
$ make push-gadget-container
The eBPF code is built using a Docker container, so you don’t have to worry installing the compilers to build it.
If you push the container images to another registry, you can use the --image
argument when deploying to the Kubernetes cluster.
Notes
- Using a locally built container image requires pushing it to a container
registry, either local or remote. The default registry can be overridden by
changing the value of the
CONTAINER_REPOenv variable, which defaults toghcr.io/inspektor-gadget/inspektor-gadgetif not defined. - The compilation uses
tools/image-tagto choose the tag of the container image to use according to the branch that you are compiling. - It is also possible to change the BCC to be used as described in BCC section.
- You can generate the required BTF information for some well known
kernel versions by setting
ENABLE_BTFGEN=true
Building the eBPF object files
If you need to compile the eBPF code of the gadgets, the ebpf-objects target will help you in this task:
$ make ebpf-objects
...
go: downloading github.com/giantswarm/crd-docs-generator v0.7.1
...
Wrote /work/pkg/gadgettracermanager/containers-map/containersmap_bpfel.go
Building ig
Inspektor Gadget also provides the
ig
tool to
trace containers without Kubernetes. It can be built independently from the
kubectl-gadget and the gadget container image.
$ make ig
Testing
Development environment on minikube
For faster iteration, it’s possible to make changes to Inspektor Gadget and test them on minikube locally without pushing container images to any registry.
- Follow the specific
installation instructions
for minikube or use
make minikube-startto start it. - Deploy the locally modified version of Inspektor Gadget to an already
running minikube cluster with
make minikube-deploy.
Unit tests
You can run the different unit tests with:
$ make test
Regenerating testdata
Some unit tests depend on precompiled files on testdata . These files can be regenerated by running
$ make testdata
Integration tests
The integration tests use a Kubernetes cluster to deploy and test Inspektor Gadget. Be sure that you have a valid kubeconfig and run:
$ export KUBECONFIG=... # not needed if valid config in $HOME/.kube/config
$ make integration-tests
Integration tests for ig
Kubernetes
The integration tests for ig uses minikube for testing different container runtimes.
The default minikube driver used for testing is docker. Currently supported
container runtimes are docker, containerd and cri-o. You can start minikube using:
$ make minikube-start-all
# for single container runtime e.g containerd
$ make CONTAINER_RUNTIME=containerd minikube-start
# for minikube driver other than docker e.g kvm2
$ make MINIKUBE_DRIVER=kvm2 minikube-start
And run the test using:
$ make -C integration/k8s test-all
# for single container runtime e.g containerd
$ make -C integration/k8s CONTAINER_RUNTIME=containerd test
if no CONTAINER_RUNTIME is specified docker will be used as a default runtime.
Non-Kubernetes
The ig integration tests for non-Kubernetes containers directly interact
with container runtime. The tests assume that you already have the desired container
runtime installed. Currently supported runtime is docker only, You can run the test using:
$ make -C integration/ig/non-k8s test-docker
Benchmarks
You can run the different benchmark tests with:
$ make gadgets-benchmarks
Or you can run an individual test with:
$ go test -exec sudo \
-bench='BenchmarkAllGadgetsWithContainers/container10$/trace-tcpconnect' \
-run=Benchmark \
./internal/benchmarks/...
Records of previous benchmarks are available here . See details in the CI documentation (benchmarks) .
Explaining performance improvements in a PR
If you want to contribute a performance improvement, it is useful to use benchmarks to explain the impact on performances. I will use the example of an improvement on the networking gadgets from #1430 :
- Run the benchmarks both on the
mainand the feature branches and saving the output in two files.
$ git checkout main
$ go test -exec sudo \
-bench='^BenchmarkAllGadgetsWithContainers$/^container100$/trace-(dns|sni)' \
-run=Benchmark \
./internal/benchmarks/... \
-count 10 | tee main.bench
$ git checkout myfeature
$ go test -exec sudo \
-bench='^BenchmarkAllGadgetsWithContainers$/^container100$/trace-(dns|sni)' \
-run=Benchmark \
./internal/benchmarks/... \
-count 10 | tee patched.bench
Please use -count to gather a statistically significant sample of results.
The
benchstat’s documentation
recommends 10 times.
- Compare the results with
benchstat:
$ go install golang.org/x/perf/cmd/benchstat@latest # if not already installed
$ benchstat main.bench patched.bench
goos: linux
goarch: amd64
pkg: github.com/inspektor-gadget/inspektor-gadget/internal/benchmarks
cpu: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
│ main.bench │ patched.bench │
│ sec/op │ sec/op vs base │
AllGadgetsWithContainers/container100/trace-dns-4 2.941 ± 3% 1.489 ± 4% -49.38% (p=0.000 n=10)
AllGadgetsWithContainers/container100/trace-sni-4 4.440 ± 19% 1.495 ± 6% -66.34% (p=0.000 n=10)
geomean 3.613 1.492 -58.72%
- Include the commands used and the output of
benchstatin your pull request description
Profiling benchmarks
You can run the different benchmark tests while using the
bcc profile tool
.
To be able to see the symbols in the profile, you need to build the binary with
-ldflags="-s=false".
$ go test -exec sudo \
-ldflags="-s=false" \
-bench='^BenchmarkAllGadgetsWithContainers$/^container100$/snapshot-socket' \
-run=Benchmark \
./internal/benchmarks/... \
-count 100
Example of output showing a stack trace including both the userspace and kernel parts:
$ sudo /usr/share/bcc/tools/profile -p $(pidof benchmarks.test)
b'established_get_first'
b'established_get_first'
b'tcp_seek_last_pos'
b'bpf_iter_tcp_batch'
b'bpf_iter_tcp_seq_next'
b'bpf_seq_read'
b'vfs_read'
b'ksys_read'
b'do_syscall_64'
b'entry_SYSCALL_64_after_hwframe'
runtime/internal/syscall.Syscall6
syscall.Syscall
syscall.read
internal/poll.(*FD).Read
os.(*File).Read
bufio.(*Scanner).Scan
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/snapshot/socket/tracer.(*Tracer).RunCollector.func1
github.com/inspektor-gadget/inspektor-gadget/pkg/netnsenter.NetnsEnter
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/snapshot/socket/tracer.(*Tracer).RunCollector
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/snapshot/socket/tracer.(*Tracer).Run
github.com/inspektor-gadget/inspektor-gadget/pkg/runtime/local.(*Runtime).RunGadget
github.com/inspektor-gadget/inspektor-gadget/internal/benchmarks.BenchmarkAllGadgetsWithContainers.func1.1
testing.(*B).runN
testing.(*B).launch
testing.(*B).doBench.func1
runtime.goexit.abi0
- benchmarks.test (3452330)
22
It is also possible to use
pprof
to profile
the benchmarks with the -cpuprofile and -memprofile flags.
go test \
-cpuprofile cpu.prof -memprofile mem.prof \
-exec sudo \
-ldflags="-s=false" \
-bench='^BenchmarkAllGadgetsWithContainers$/^container100$/snapshot-socket' \
-run=Benchmark ./internal/benchmarks/... \
-count 5
$ go tool pprof -top cpu.prof
$ go tool pprof -top mem.prof
Continuous Integration
Inspektor Gadget uses GitHub Actions as CI. Please check dedicated CI documentation for more details.
Some integration tests (like AKS and ARO) are only run when a commit is pushed to the main branch or
a new tag is pushed. It’s also possible to run those by pusing a branch named citest/.... Please
notice that the container images will be pushed to
https://github.com/inspektor-gadget/inspektor-gadget/pkgs/container/inspektor-gadget
and those
should be manually cleaned up.
Getting Help
If you are having any issues with your contribution, or want to discuss anything about it, feel free to reach out to us on Slack or on our community meeting
Contribution Guidelines
Code of Conduct
Inspektor Gadget follows the CNCF Code of Conduct .
Authoring PRs
For making PRs/commits consistent and easier to review, please check out Kinvolk’s contribution guidelines on git .
We require all commits on a PR to be signed off certifying the Developer Certificate of Origin :
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
It can be done by using git commit -s, git commit --sign-off or by manually adding a line like
to the commit message.
Signed-off-by: Joe Smith <[email protected]>
Good first issues
If you’re looking where to start, you can check the issues with the
good first issue label on
Inspektor Gadget
.
Don’t hesitate to
talk to us
if you need further help.
Proposing new features
If you want to propose a new feature or do a big change in the architecture it’s highly recommended to open an issue first to discuss it with the team.
Writing tests
We use github.com/stretchr/testify to make tests less verbose.
Planning
Our planning is published through two different project boards:
- Inspektor Gadget Roadmap has the high level view of the big issues that we are planning to tackle in the upcoming months.
- Inspektor Gadget Sprint Planning has the week-to-week plans of which bugs we are currently working on, and the different priorities of the issues involved.
BCC
Porting BCC gadgets
This project uses some gadgets based on BCC . In the past, we modified the BCC gadgets and executed them from our process, however it was very inflexible and we decided to integrate those gadgets directly into our code base by rewriting their control plane in Golang.
If you want to implement support for a BCC gadget, please read the Rewriting the Control Plane of BCC Tools in Golang blogpost that contains all the details about this process.
Security
For security, we invite you to take at look at the dedicated document .