Disk filling up due to etcd backups in a kURL cluster

After multiple upgrades of a kURL cluster, you might get to a point where the partition used by kubernetes, usually root, starts filling up. After some troubleshooting, you might see loads of files in /etc/kubernetes/tmp directory named either kubeadm-backup-etcd-<date>-<time> or kubeadm-backup-manifests-<date>-<time>. These are backups stored by kubeadm upgrade command which the kURL installer uses internally.

  • kubeadm-backup-manifests-<date>-<time>: Static pod manifests. These are fairly small in size (<1 KB)
  • kubeadm-backup-etcd-<date>-<time>: etcd database snapshots backed up just before running kubeadm upgrade. These can end up being large directories especially if the cluster has many resources. Each control plane node in the cluster will have these backups. These are the directories you most likely want to clean up.

NOTE: When deleting etcd backups, ensure you leave at least one backup behind for recovery if needed

To view all the backups you have in a node, you would run

$ sudo ls -al /etc/kubernetes/tmp/
total 20
drwx------ 5 root root 4096 Sep  5 13:44 .
drwxr-xr-x 5 root root 4096 Sep  5 14:19 ..
drwx------ 3 root root 4096 Sep  5 13:42 kubeadm-backup-etcd-2023-09-05-13-42-31
drwx------ 2 root root 4096 Sep  5 13:44 kubeadm-backup-manifests-2023-09-05-13-42-31
drwx------ 2 root root 4096 Sep  5 13:44 kubeadm-kubelet-config388725400

References