Replicated upgrade from 2.38.0 to 2.49.0 leaves kubeadm and kubelet at version 1.15.3

Supporting your customers
1

Hey There,
We recently upgraded replicated to v2.49.0. This version is supposed to upgrade Kubernetes from v.1.15.3 to v.1.15.12 and it did.
After the upgrade, we noticed that the Kubernetes server is upgraded to v.1.15.12 but kubeadm and kubelet are still left at v.1.15.3. I went through the install script and found that we are force upgrading kubeadm to v.1.15.12 with kubeadm upgrade apply "v$k8sVersion" --yes --config /opt/replicated/kubeadm.conf --force and then we are replacing kubeadm binary from archives folder.

Even I tried upgrading it manually by running kubeadm upgrade apply v.1.15.12 it returns back [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.12". Enjoy! but kubeadm version still returns v.1.15.3.

Any ideas why kubeadm and kubectl were left out and how to get it upgraded?
In case I need to upgrade kubeadm binary manually how can that be done? I couldn’t find the binaries anywhere on the internet. Should I compile and get it replaced?

2

Did you use the kubernetes-upgrade-patch-version flag?

3

Nope, I was informed to re-run the script from the docs. Let me try that. Thanks.

4

For reference, optional installer flags can be found here

https://help.replicated.com/docs/kubernetes/customer-installations/installing/

kubernetes-upgrade-patch-version

Upgrade to the latest available patch release of Kubernetes
5

@areed, @ethanm
The upgrade gets stuck for hours waiting for rook/ceph’s health report. Here you go with the screenshot.

image
image721×367 42.1 KB

Rook doesn’t come up. When I tried checking docker containers using docker ps -a all I can see is ceph containers(k8s_envoy-initconfig_contour*, k8s_init-mon-fs_rook-ceph-mon-a-*) getting exited and there are no logs for them.

6

Can you get the ceph status or a support bundle?

http://community.replicated.com/t/kots-how-can-i-generate-a-support-bundle-if-i-cannot-access-the-admin-console

kubectl -n rook-ceph exec $(kubectl -n rook-ceph get pods -l app=rook-ceph-operator --no-headers | awk '{print $1}') -- ceph status
7

@ethanm Thanks for the help. I’ve fixed the issue by manually downloading kubelet and kubeadm binaries and replacing them.