If you have recently upgraded your Embedded Cluster to a version running Kubernetes v1.31, you may encounter a startup issue on controller nodes where pods fail to initialize correctly.
Symptoms
A review of the kubelet logs on the affected controller will typically show repeated instances of this error:
Error: services have not yet been read at least once, cannot construct envvars
Root Cause
This issue stems from a race condition that can occur on a controller node’s first boot after the Embedded Cluster upgrade.
In some cases, the kubelet service starts before the kube-apiserver is fully initialized. When this happens, the kubelet fails to retrieve the cluster’s service information, which is necessary for injecting service-related environment variables (e.g., *_SERVICE_HOST, *_SERVICE_PORT) into new pods. This causes any pod that relies on these variables to fail.
Workaround
A straightforward workaround is to manually restart the k0s controller service after the node has completed its upgrade. This allows the components to re-initialize in the correct order, ensuring the API server is ready before the kubelet begins creating pods.
On the affected controller node, run the following command:
sudo systemctl restart k0scontroller
This should resolve the race condition and allow the pods to start successfully.
NOTE: If restarting the k0scontroller service doesn’t resolve the issue, you may need to restart the node entirely. After restarting the node, be sure to delete any pods which are stuck in the 0/1 Completed status.
This issue is being tracked on the official k0s GitHub repository: https://github.com/k0sproject/k0s/issues/6071.