trace_fsslower
The trace fsslower gadget streams file operations (open, read, write and fsync) that are slower than a threshold.
Requirements
- Minimum Kernel Version : 5.4
Getting started
Running the gadget:
- kubectl gadget
- ig
$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/trace_fsslower:v0.58.0 [flags]
$ sudo ig run ghcr.io/inspektor-gadget/gadget/trace_fsslower:v0.58.0 [flags]
Flags
--min
Minimum latency in µs to trace
Default value: "10000"
--pid
Show only bind events generated by this particular PID
Default value: ""
--filesystem
Filesystem to trace. Possible values are: btrfs, ext4, fuse, nfs, nfts3 or xfs.
Default value: "ext4"
Guide
In this guide you'll deploy an example workload that performs some open(), read() write() and sync() calls and will trace which ones are slower than 1 ms.
Let's start the gadget before running our workload:
- kubectl gadget
- ig
$ kubectl gadget run trace_fsslower:v0.58.0 --podname test-trace-fsslower --filesystem ext4 --min 1000
K8S.NODE K8S.NAMESPACE K8S.PODNAME K8S.CONTAINERNAME COMM PID TID TID OFFSET FILE OP SIZE DELTA
$ sudo ig run trace_fsslower:v0.58.0 --verify-image=false --filesystem ext4 --min 1000 -c test-trace-fsslower
RUNTIME.CONTAINERNAME COMM PID TID OFFSET FILE OP SIZE DELTA
Launch a container that will perform input/output operations:
- kubectl gadget
- ig
$ kubectl run test-trace-fsslower --restart=Never --image=debian -- /bin/sh -c "apt-get update && apt-get install -y git"
pod/test-trace-fsslower created
$ docker run --name test-trace-fsslower -it --rm debian /bin/sh -c "apt-get update && apt-get install -y git"
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
The tool will list the I/O operations that were slower than 1ms:
- kubectl gadget
- ig
$ kubectl gadget run trace_fsslower:v0.58.0 --podname test-trace-fsslower --filesystem ext4 --min 1000
K8S.NODE K8S.NAMESPACE K8S.PODNAME K8S.CONTAINERNAME COMM PID TID TID OFFSET FILE OP SIZE DELTA
minikube default test-trace-fsslower test-trace-fsslower http 60778 60778 60775 3857488 deb.debian.org_… F_WRITE 8.5 kB 5.83ms
minikube default test-trace-fsslower test-trace-fsslower http 60778 60778 60775 6647227 deb.debian.org_… F_WRITE 1.7 kB 1.39ms
minikube default test-trace-fsslower test-trace-fsslower store 60786 60786 60775 14163567 deb.debian.org_… F_WRITE 26 kB 2.08ms
minikube default test-trace-fsslower test-trace-fsslower dpkg-preconfigu 60898 60898 60897 0 #56749 F_FSYNC 9.2 EB 3.08ms
minikube default test-trace-fsslower test-trace-fsslower dpkg-preconfigu 60898 60898 60897 0 #56751 F_FSYNC 9.2 EB 2.46ms
minikube default test-trace-fsslower test-trace-fsslower dpkg-preconfigu 60898 60898 60897 0 config.dat-new F_FSYNC 9.2 EB 3.39ms
minikube default test-trace-fsslower test-trace-fsslower dpkg-preconfigu 60898 60898 60897 0 templates.dat-n… F_FSYNC 9.2 EB 2.08ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 triggers F_FSYNC 9.2 EB 3.67ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 shlibs F_FSYNC 9.2 EB 3.03ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 symbols F_FSYNC 9.2 EB 3.35ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 md5sums F_FSYNC 9.2 EB 3.06ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 control F_FSYNC 9.2 EB 3.11ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 tmp.ci F_FSYNC 9.2 EB 2.27ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 tmp.i F_FSYNC 9.2 EB 1.84ms
minikube default test-trace-fsslower test-trace-fsslower dpkg 60985 60985 60849 0 updates F_FSYNC 9.2 EB 1.57ms
$ sudo ig run trace_fsslower:v0.58.0 --verify-image=false --filesystem ext4 --min 1000 -c test-trace-fsslower
RUNTIME.CONTAINERNAME COMM PID TID OFFSET FILE OP SIZE DELTA
test-trace-fsslower dpkg-preconf… 204239 204239 0 #38586681 F_FSYNC 9.2 EB 1.01ms
test-trace-fsslower dpkg-preconf… 204239 204239 0 #38586683 F_FSYNC 9.2 EB 1.88ms
test-trace-fsslower dpkg 204242 204242 0 perl-module… F_FSYNC 9.2 EB 3.25ms
test-trace-fsslower dpkg 204242 204242 0 libperl5.36… F_FSYNC 9.2 EB 1.52ms
test-trace-fsslower dpkg 204242 204242 0 symbols F_FSYNC 9.2 EB 1.27ms
test-trace-fsslower dpkg 204242 204242 0 openssl.lis… F_FSYNC 9.2 EB 1.28ms
test-trace-fsslower dpkg 204242 204242 0 ca-certific… F_FSYNC 9.2 EB 1.11ms
test-trace-fsslower dpkg 204242 204242 0 tmp.i F_FSYNC 9.2 EB 1.31ms
test-trace-fsslower dpkg 204242 204242 0 triggers F_FSYNC 9.2 EB 6.30ms
test-trace-fsslower dpkg 204242 204242 0 tmp.i F_FSYNC 9.2 EB 1.44ms
test-trace-fsslower dpkg 204242 204242 0 libkrb5supp… F_FSYNC 9.2 EB 1.04ms
test-trace-fsslower dpkg 204242 204242 0 md5sums F_FSYNC 9.2 EB 1.32ms
test-trace-fsslower dpkg 204242 204242 0 openssh-cli… F_FSYNC 9.2 EB 1.03ms
test-trace-fsslower dpkg 204242 204242 0 symbols F_FSYNC 9.2 EB 1.06ms
test-trace-fsslower dpkg 204242 204242 0 git.list-new F_FSYNC 9.2 EB 2.53ms