In the case of an air gapped installation, the kURL registry is used to store the application images. The registry add-on has the capability to garbage collect images that are no longer in use, and KOTS triggers that capability in certain scenarios.
For example, when upgrading to an application version that does not use certain images anymore, those images will be removed from the registry if:
1- The application does not support rollbacks.
2- The images are not listed in the additionalImages section of the KOTS application spec.
3- Image garbage collection was not disabled manually.
However, there is a known bug in the registry add-on where if an image gets garbage collected, it can no longer be used even if it was pushed again.
Solution
1- Restart the registry pods:
kubectl rollout restart deploy/registry -n kurl
2- Re-push the image(s) from the airgap bundle using the CLI:
kubectl kots admin-console push-images \
</path/to/bundle.airgap> \
$(kubectl get svc registry -n kurl -o jsonpath='{.spec.clusterIP}')/<app-slug> \
--registry-username kurl \
--registry-password $(kubectl get secret registry-creds -o=jsonpath='{.data.\.dockerconfigjson}' | base64 -d | grep -Po '"password"\s*:\s*"\K[^"]+')
Replace </path/to/bundle.airgap> with the path to the airgap bundle.
Replace <app-slug> with the application slug.
Now, to prevent the issue from happening in the future, you can do one of the following:
- Add the images to additionalImages in the KOTS application spec so they will not be part of the image garbage collection.
- Disable image garbage collection. Reference: Image Registry for Embedded kURL Clusters | Replicated Docs
- Enable rollback so that image garbage collection never runs.