StatefulSet pods stuck in Completed after node restart

Whenever pods part of a StatefulSet are gracefully terminated, commonly observed during reboots, its possible for them to be left in a state of Completed which will cause other applications running in the cluster relying on those services to fail to start.

This appears to be a regression in Kubernetes 1.27 and the issue is being tracked here - StatefulSet pod ends up in state "Completed" · Issue #124065 · kubernetes/kubernetes · GitHub.

To recover from this you can simply delete the Completed pod so the StatefulSet controller will create a new pod.

kubectl delete pod --field-selector=status.phase=Succeeded --all

In the upstream issue its mentioned that this regression is fixed in recent patch versions. For 1.27 it was fixed with 1.27.9