Determine disk usage by volume

Supporting your customers
1

PVCs

Save and run this script as root to determine how much storage each PVC is consuming.

#!/bin/bash

cd /var/lib/kubelet/pods

while read -r line; do
        relpath=$(echo $line | awk '{ print $2 }')
        pv=$(basename $relpath)
        pvc=$(KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pv $pv --all-namespaces --no-headers | awk '{ print $6 }')
        size=$(echo $line | awk '{ print $1 }')

        echo "$size     $pvc"
done < <(find . -type d -name 'pvc-*' | xargs du -sh | sort -hr)

Empty Directories

Save this script and run it as root to determine disk usage for each empty directory volume:

#!/bin/bash

export KUBECONFIG=/etc/kubernetes/kubelet.conf

allPodUIDs=$(kubectl get pods --all-namespaces -ojsonpath='{ range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\t"}{.metadata.namespace}{"\n"}{end}' )

while read -r line; do
        uid=$(echo "$line" | grep -Eo "pods\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}" | sed 's/pods\///')
        pod=$(echo "${allPodUIDs[*]}" | grep "$uid")
        usage=$(echo "$line" | awk '{ print $1 }')

        echo "$usage $pod"
done < <(find /var/lib/kubelet/pods -name '*empty-dir' | xargs du -sh | sort -hr)

Container Layer

Pods may also be writing excess data to a directory that does not have any volume mounted. In this case the data will be written the container’s ephemeral writable layer. Running the docker system df command in the example below shows that 2GB total has been written to docker container layers.

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          35        30        4.787GB   1.882GB (39%)
Containers      89        71        2GB       0B (0%)
Local Volumes   2         2         0B        0B
Build Cache     0         0         0B        0B

If you’re using docker with overlay2 as your CRI, you can save this script and run it as root to identify the 10 containers that have written the most data to their ephemeral container layer:

#!/bin/bash

function print() {
        while read -r container; do
                name=$(echo $container | awk '{ print $NF }')
                usage=$(du -sh $(docker inspect --format='{{ .GraphDriver.Data.UpperDir }}' $name))
                echo "$usage $name"
        done < <(docker ps | grep -v pause | sed '1d')
}

print | sort -hr | head