Migrate etcd to New Mount Point (kURL Clusters)

1

Note: This guide is specifically for kURL clusters only.

Prerequisites

  • New disk attached, partitioned, formatted, and ready to mount

  • Access to the control plane node

  • Cluster has multiple control plane nodes (or plan for downtime)

Migration Steps

1. Shutdown EKCO and drain the node

# Run EKCO shutdown script
sudo /opt/ekco/shutdown.sh

# Drain the node (run from a control plane node or bastion)
kubectl drain <node-name> --ignore-daemonsets --timeout=5m

2. Stop etcd by moving manifests

# Move manifests to stop static pods
sudo mv /etc/kubernetes/manifests .

# Wait for etcd to stop (verify pod is gone)
# ps aux | grep etcd  # should return nothing

3. Backup existing etcd data

# Create backup of etcd data
sudo cp -a /var/lib/etcd/ /var/lib/etcd.backup

# Verify backup
ls -la /var/lib/etcd.backup

4. Prepare and mount new disk to /var/lib/etcd

Work with your sysadmin to:

  • Move the old /var/lib/etcd directory (e.g., to /var/lib/etcd.old)

  • Create a new /var/lib/etcd mount point

  • Mount the new disk to /var/lib/etcd

  • Add the mount to /etc/fstab to persist across reboots

  • Verify the mount is successful with df -h | grep etcd

5. Restore etcd data to new mount

# Copy data from backup to new mount point
sudo cp -aR /var/lib/etcd.backup/. /var/lib/etcd/

# Verify data copied correctly
ls -la /var/lib/etcd

6. Restore manifests and wait for etcd

# Move manifests back to start etcd
sudo mv manifests /etc/kubernetes/.

# Wait for etcd pod to start and cluster to be ready
watch kubectl get nodes

# Verify etcd is running
crictl ps | grep etcd
kubectl get pods -n kube-system | grep etcd

7. Restart the node

# Reboot to trigger EKCO startup
sudo reboot

# After reboot, EKCO startup script will run automatically
# /opt/ekco/startup.sh will be executed

8. Post-restart verification

# Check node status
kubectl get nodes

# Uncordon if needed
kubectl uncordon <node-name>

# Verify etcd mount persisted
df -h | grep etcd
mount | grep etcd

# Check etcd health
kubectl get pods -n kube-system | grep etcd

# Clean up old data once verified
sudo rm -rf /var/lib/etcd.old /var/lib/etcd.backup

Important Notes

  • For HA clusters: If the cluster is HA, perform this change on each node one at a time. Workloads will be moved from one node to other nodes after the node has been cordoned. The Kubernetes cluster should still be healthy but your applications might not be depending on how they handle nodes being shutdown.

  • For single control plane clusters: The whole cluster will be unavailable during this migration.

  • EKCO scripts handle cluster-specific startup/shutdown procedures

  • Verify Kubernetes cluster health between each node migration

  • Keep backups until you’ve verified the cluster is stable

  • Coordinate with your sysadmin for disk preparation, partitioning, and formatting

Troubleshooting

If etcd fails to start after migration:

# Check etcd logs
sudo journalctl -xeu kubelet | grep etcd
sudo crictl ps | grep etcd  # find etcd container id
sudo crictl logs <etcd-container-id>

# Verify data ownership
sudo ls -la /var/lib/etcd

# Verify mount
mount | grep etcd