Resolving Unexpected Image rewrites in HelmChart v1beta1

1

Resolving Unexpected Image rewrites in HelmChart v1beta1

Understanding the Failure

If you’re using HelmChart v1beta1 in your application, you may encounter pods failing to start due to image pull errors. These errors might appear as “ImagePullBackOff” or “ErrImagePull” in your pod status. With v1beta1 it’s possible this is happening because of unintended rewrites of images to use the replicated proxy.

KOTS Automatic Image Rewrite

When KOTS encounters difficulties pulling images (often due to rate limiting), it automatically attempts to rewrite the image URL in the podspec. This rewrite occurs only for HelmChart v1beta1 and results in the image URL being prefixed with proxy.replicated.com.

Confirming the Issue

To confirm if this automatic rewrite is affecting your deployment:

  1. Check the image URL in your pod specification:
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].image}'
  1. If the image URL starts with proxy.replicated.com/proxy/<app-slug>/ and you did not intend for it to then you’re experiencing this issue.

Resolving the Issue

If you’ve confirmed that KOTS has rewritten your image URL unexpectedly you first need to confirm the original cause of the failed pull is resolved. If this has happened something caused KOTS to fail to pull, which is why it is trying to rewrite now. Common causes are hitting a rate limit from your upstream repository or intermittent connectivity failures. Confirm you can now pull from the expected upstream before fixing the unexpected rewrite or KOTS will simply rewrite it again when it fails to pull.

Once the original issue is resolved, follow these steps to return to the expected upstream without the proxy:

  1. Access your application’s Admin Console.

  2. Navigate to the “Config” tab.

  3. Make a temporary configuration change. This should be a minor alteration to some config value.

  4. Save your changes.

  5. Deploy the new configuration by clicking the “Deploy” button in the Admin Console.

  6. This deployment will trigger KOTS to regenerate the manifests, reverting the image URL to its original value.

  7. Once the deployment is complete, verify the change by checking the pod specification again:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].image}'
  1. If the image URL has been corrected, you can revert the temporary change if desired by repeating steps 1-5.

This automatic rewrite only happens with v1beta1, for a more permanent solution please upgrade your application to v1beta2. Documentation on making that migration can be found at: Configuring the HelmChart Custom Resource v2 | Replicated Docs

1 Like