I have attempted to upgrade a KOTS node (single-node embedded cluster) by re-running the kURL script, as described on Updating an Embedded Cluster - Kots.io. The node was being upgraded from installer-1.yaml to installer-2.yaml.
# installer-1.yaml
apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
name: ""
spec:
contour:
version: "1.19.x"
certManager:
version: "1.0.x"
docker:
version: "20.10.x"
ekco:
version: "0.13.x"
kotsadm:
version: "1.59.x"
kubernetes:
version: "1.21.x"
longhorn:
version: "1.2.x"
minio:
version: "2020-01-25T02-50-51Z"
prometheus:
version: "0.49.x"
registry:
version: "2.7.x"
weave:
version: "2.6.x"
velero:
version: "1.7.x"
# installer-2.yaml
apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
name: ""
spec:
contour:
version: "1.19.x"
docker:
version: "20.10.x"
ekco:
version: "0.16.x"
kotsadm:
version: "1.62.x"
kubernetes:
version: "1.22.x"
longhorn:
version: "1.2.x"
uiBindPort: 30880
uiReplicaCount: 1
minio:
version: "2020-01-25T02-50-51Z"
prometheus:
version: "0.53.x"
registry:
version: "2.7.x"
weave:
version: "2.6.x"
velero:
version: "1.7.x"
I ran into an issue where the script performed most of its steps, but exited with errors near the end:
rolebinding.rbac.authorization.k8s.io/kurl-proxy-tls-secrets unchanged
service/kurl-proxy-kotsadm unchanged
deployment.apps/kurl-proxy-kotsadm unchanged
[|]
Message from syslogd@ip-172-16-4-148 at Feb 8 16:02:15 ...
kernel:EXT4-fs (sdh): failed to convert unwritten extents to written extents -- potential data loss! (inode 792986, error -30)
Message from syslogd@ip-172-16-4-148 at Feb 8 16:02:15 ...
kernel:EXT4-fs (sdh): failed to convert unwritten extents to written extents -- potential data loss! (inode 790386, error -30)
Message from syslogd@ip-172-16-4-148 at Feb 8 16:02:15 ...
kernel:EXT4-fs (sdh): failed to convert unwritten extents to written extents -- potential data loss! (inode 791251, error -30)
Error from server (BadRequest): container "restore-s3" in pod "kotsadm-79f7b7ff7-hprrz" is waiting to start: PodInitializing
An error occurred on line
Would you like to provide a support bundle to aid us in avoiding similar errors in the future?
(y/N)
From my observations, the script was waiting for all containers in the kotsadm pod to become healthy, but timed out before they were healthy.
After the script exited, eventually kotsadm finished booting and became healthy.
The ext4 errors are also concerning, though I’m not sure if or how they are related.
Executing the script again results in the same errors as above, even after rebooting the server and adding more CPUs/memory.
Are there any additional steps I can take to debug this issue?