Troubleshoot Replicated Proxy Service

Troubleshooting
1

This guide will help you troubleshoot issues related to the Replicated Proxy Service. Follow these steps to identify and resolve common problems.

1. Verify Registry Configuration

First, ensure that the registry endpoint is correctly configured via Vendor Portal or with Replicated CLI

replicated registry ls

Expected output:

PROVIDER     ENDPOINT                                             AUTHTYPE
ecr          193516486740.dkr.ecr.ap-southeast-2.amazonaws.com    accesskey

If the output doesn’t show the expected registry, review your Replicated configuration.

2. Test Registry Authentication

Verify that the authentication configuration is correct:

replicated registry test <registry> --image <image>

Successful authentication:

Registry connection appears ok

Failed authentication:

Registry connection failed with status 401

3. Inspect Image from Private Registry

Use skopeo or your preferred container images CLI to inspect the image directly from the private registry:

skopeo inspect docker://193516486740.dkr.ecr.ap-southeast-2.amazonaws.com/<image> | jq

Sample output:

{
  "Name": "193516486740.dkr.ecr.ap-southeast-2.amazonaws.com/gerard",
  "Digest": "sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0",
  "RepoTags": [
    "latest"
  ],
  "Created": "2024-01-27T00:30:48.743965523Z",
  "DockerVersion": "20.10.23",
  "Labels": null,
  "Architecture": "amd64",
  "Os": "linux",
  "Layers": [
    "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8"
  ],
  "LayersData": [
    {
      "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "Digest": "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8",
      "Size": 3408729,
      "Annotations": null
    }
  ],
  "Env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  ]
}

Ensure that the required image tag is available in the JSON output.

4. Test image pull via Replicated Proxy

Finally, test pulling the image by connecting to the Replicated Proxy directly.

Get the registry credential username:password from existing secret in Kubernetes. The secret should be of type kubernetes.io/dockerconfigjson with naming convention of <app-slug>-registry

export APP_SLUG=<your_app_slug>
export REGISTRY=<proxy.replicated.com OR custom domain>
export CREDS=$(kubectl get secret $APP_SLUG-registry -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r ".auths[\"$REGISTRY\"].auth" | base64 -d)

Test image pull with ctr

sudo ctr images pull -u $CREDS $REGISTRY/proxy/$APP_SLUG/<image:tag>

Successful pull sample output:

proxy.replicated.com/proxy/<app-slug>/193516486740.dkr.ecr.ap-southeast-2.amazonaws.com/<image:tag>: resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:b99dff2e6b5e90260534efc6eb8bda6001859cb350e6c24067c505616a61afcf:                                done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:6e32ae02d1e8c4aa6a8c7439c9cd430db50bc38ac7aff32ebc87c8ca36272ac1:                                  done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:cbc47022ee338ad066c7bafa3d0647a8104564403053dd9e624a970a6099bf4b:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f1c1bc08e5d6f5cdcd4a360e22c4eb32353dba076fd66cf0bfab630b240fe34f:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f2184fe5ad83d7f98704f5183454666c6cd05babea1f8ed9cd0df84dae2c41d:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:344197a120544316a0a53414d55e7485b4c58ecbe3e92ca7a3f4f532eafc9e40:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1558431db48b9fa26098243bff8b8f8e02c05efd7027dee46580daa823b9d6bf:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c999b5bed5bfb843496c389c1e5564f6f4d32f6b226698e4b4498265f5d99e12:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f6b6ea6fa830cc548cce4e367920e1d2f097506b35f439b33f45340699c005de:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:68554efeeace362e2063b001923a87c77ea467cf46e76d21d6b04ce12d8273e2:                                   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5bacf9fa5da05433596827d36351bb69194de4387cc926c8824eeff67191c873:                                   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 10.6s                                                                                                   total:  46.0 M (4.3 MiB/s)
unpacking linux/amd64 sha256:b99dff2e6b5e90260534efc6eb8bda6001859cb350e6c24067c505616a61afcf...
done: 4.18518749s

Alternatively, use skopeo to download the image locally:

skopeo copy --src-creds $CREDS docker://$REGISTRY/proxy/$APP_SLUG/<image:tag> docker-archive:<image_name>

Successful download sample output:

Getting image source signatures
Copying blob c158987b0551 done   |
Copying blob 68554efeeace done   |
Copying blob c999b5bed5bf done   |
Copying blob cbc47022ee33 done   |
Copying blob 0f2184fe5ad8 done   |
Copying blob 1558431db48b done   |
Copying blob f6b6ea6fa830 done   |
Copying blob f1c1bc08e5d6 done   |
Copying blob 344197a12054 done   |
Copying blob 5bacf9fa5da0 done   |
Copying config 6e32ae02d1 done   |
Writing manifest to image destination

If you encounter persistent issues after following these steps, please contact Replicated support and provide the outputs from these commands for further assistance.