I'm running into problems with install bundles for airgapped clusters

How do I?
,
1

Hi,
I am encountering problems when attempting to update versions on airgapped kURL clusters if the new version needs to update by more than 1 Kubernetes version. I know that an installer bundle contains only one version of each add-on. I created additional bundles to update the Kubernetes version iteratively, but the problem I’m having is that only the kubeadm gets updated. The kubelet and kubectl don’t.

The first problem I had was that I took the instructions literally (I don’t actually know that being literal was the problem, but it seems likely.):

⚙  Downloading assets required for Kubernetes 1.30.7 to 1.33.4 upgrade
The following packages are not available locally, and are required:
kubernetes-1.31.14.tar.gz
kubernetes-1.32.12.tar.gz

You can download them with the following command:

curl -LO https://kurl.sh/bundle/version/v2026.02.24-0/c9f92c5/packages/kubernetes-1.31.14,kubernetes-1.32.12.tar.gz

So instead I created installation bundles that would only change the Kubernetes version, iteratively, and ran them. But I encountered messages that looked like this:

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.4".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
⚙  Install kubelet, kubectl and cni host packages
⚙  Installing host packages kubelet-1.33.4 kubectl-1.33.4
kURL kubernetes-1.33.4 Local Repo               414 kB/s | 3.9 kB     00:00
All matches were filtered out by exclude filtering for argument: kubelet-1.33.4
All matches were filtered out by exclude filtering for argument: kubectl-1.33.4
Error: Unable to find a match: kubelet-1.33.4 kubectl-1.33.4
An error occurred on line 3084

I was unable to just run the install command again because the pods in the kurl namespace (etcd and the two registry pods) were stuck in Pending.

What allowed me to proceed was to manually update the kubelet by this procedure:
Download the kubelet with a command like this (on a different machine)
curl -LO https://dl.k8s.io/v1.33.4/kubernetes-server-linux-amd64.tar.gz
then after pushing the tarball to the airgapped VM, I ran the following commands:

tar -xzf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
sudo systemctl stop kubelet
sudo mv /usr/bin/kubelet /usr/bin/kubelet-backup
sudo cp kubelet /usr/bin/
sudo systemctl daemon-reexec
sudo systemctl start kubelet
sudo systemctl enable kubelet
kubelet --version
kubectl uncordon <nodeName>
kubectl rollout restart sts -n knime
kubectl rollout restart deployments -n knime

That allowed me to proceed, but even so, I ended up with the versions of kubeadm, kubelet, and kubectl not matching (at least insofar as the client version of kubectl was still old).
So then I separately downloaded kubectl on a separate machine:
curl -LO "https://dl.k8s.io/release/v1.33.4/bin/linux/amd64/kubectl"
And then pushed that to the airgapped VM, and moved kubectl to /usr/bin (/usr/local/bin didn’t work).

So my question is how do I get this to work correctly, airgapped, on RHEL? (I don’t know that RHEL is any different from Ubuntu, but I don’t recall having this particular problem in the past.)

2

I was able to work through it, eventually, but this is not as clean as I would like for my customers.

3

@lepome I’m glad to hear you managed to get past the upgrade hurdle. Could you let me know what part in the upgrade flow you’d like to see reduced friction? You can paste commands from the latest successful upgrade steps you ran. Also, feel free to raise a support ticket in support.replicated.com if you would like to share more sensitive details.

4

The commands that allowed me to complete the update are those I provided. I would have preferred, however, if I could have skipped the parts where I got the error message and then iterative steps downloading kubelet for each version, and also the separate download and push of kubectl.

I would have preferred if the multiple versions of Kubernetes could be installed with a single download and push, and if that’s not possible, then with a single download and push for each version. Having to download kubelet and kubectl separately was not documented. (Note that this kind of process is necessary when Airgapped instances are updated from older versions because the default installer from the download portal is only the very latest one, so neither does it include multiple versions of Kubernetes, which must be iteratively updated, nor does it include the older versions of other packages.)