If you have recently upgraded your Embedded Cluster to a version running Kubernetes v1.31, you may encounter a startup issue on controller nodes where pods fail to initialize correctly.
Symptoms
A review of the kubelet logs on the affected controller will typically show repeated instances of this error:
Error: services have not yet been read at least once, cannot construct envvars
Root Cause
This issue stems from a race condition that can occur on a controller node’s first boot after the Embedded Cluster upgrade.
In some cases, the kubelet service starts before the kube-apiserver is fully initialized. When this happens, the kubelet fails to retrieve the cluster’s service information, which is necessary for injecting service-related environment variables (e.g., *_SERVICE_HOST, *_SERVICE_PORT) into new pods. This causes any pod that relies on these variables to fail.
Workaround
A straightforward workaround is to manually restart the k0s controller service after the node has completed its upgrade. This allows the components to re-initialize in the correct order, ensuring the API server is ready before the kubelet begins creating pods.
On the affected controller node, run the following command:
sudo systemctl restart k0scontroller
This should resolve the race condition and allow the pods to start successfully. This issue is being tracked on the official k0s GitHub repository: https://github.com/k0sproject/k0s/issues/6071.