Using trace fsslower

    Screencast of the trace fsslower gadget

    The trace fsslower gadget streams file operations (open, read, write and fsync) that are slower than a threshold.

    On Kubernetes

    In this guide you’ll deploy an example workload that performs some open(), read() write() and sync() calls and will trace which ones are slower than 1 ms.

    Let’s start the gadget before running our workload:

    $ kubectl gadget trace fsslower -f ext4 -m 1 -p mypod
    NODE             NAMESPACE        POD              CONTAINER        PID     COMM             T BYTES  OFFSET  LAT      FILE
    

    With -f we’re indicating the type of filesystem we want to trace, ext4 in this case. The -m parameter indicates the threshold, in this case operations taking more than 1ms will be printed. -p indicates that we only want to trace events coming from mypod.

    The T column indicates the operation type, O for open, R for read, W for write and F for fsync.

    In another terminal, let’s create a pod that updates the apt-get cache and installs git.

    $ kubectl run -it mypod --image ubuntu -- /bin/sh -c "apt-get update && apt-get install -y git"
    ...
    

    We can see how fsslower shows the operations that are taking longer than 1ms:

    $ kubectl gadget trace fsslower -f ext4 -m 1 -p mypod
    NODE             NAMESPACE        POD              CONTAINER        PID     COMM             T BYTES  OFFSET  LAT      FILE
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       2.66     perl-modules-5.30.list-new
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.49     libperl5.30:amd64.list-new
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.45     control
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.01     less.list-new
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.05     symbols
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.05     md5sums
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.16     control
    ubuntu-hirsute   default          mypod            mypod            579778  dpkg             F 0      0       1.09     git.list-new
    ubuntu-hirsute   default          mypod            mypod            580362  dpkg             F 0      0       1.16     tmp.i
    ubuntu-hirsute   default          mypod            mypod            580363  frontend         F 0      0       1.50     templates.dat-new
    ubuntu-hirsute   default          mypod            mypod            582040  dpkg-trigger     F 0      0       1.10     triggers
    ubuntu-hirsute   default          mypod            mypod            580382  frontend         F 0      0       1.22     templates.dat-new
    ubuntu-hirsute   default          mypod            mypod            583411  dpkg             F 0      0       2.25     perl-modules-5.30.list-new
    ubuntu-hirsute   default          mypod            mypod            583411  dpkg             F 0      0       2.05     libperl5.30:amd64.list-new
    ubuntu-hirsute   default          mypod            mypod            583411  dpkg             F 0      0       1.13     tmp.i
    ubuntu-hirsute   default          mypod            mypod            583411  dpkg             F 0      0       1.26     updates
    ubuntu-hirsute   default          mypod            mypod            583411  dpkg             F 0      0       1.22     md5sums
    

    That’s all, let’s delete our example pod

    $ kubectl delete pod mypod
    

    With local-gadget

    TODO