Posting here because I bet this is of wider interest or experience than just us, and I bet this will start happening more.
We’re installing into the IBM public cloud and hitting docker.io rate limits pulling KOTs images, eg docker.io/kotsadm/minio:0.20231220.010002-r2 etc, even if we manually pull with docker CLI there. One hypothesis is IBM is routing through a small number of egress IPs and all those requests count against rate limits at Docker.
As a workaround in a “brown listed” environment like that, it seems like you could use the private registry airgap procedure to install KOTs and then use the regular online install which will pull from proxy.replicated.com … yes?
Longer term though, would it make sense to move the KOTs images off docker.io?
Hey, @Mitch. My recommendation today is to set proxyPublicImages: true in your kind: Application. This field isn’t currently documented, but we’re going to add docs for it. When KOTS sees this, it will proxy public images in addition to private ones. Add Docker Hub credentials to the Vendor Portal (docs), and then the Docker Hub rate limit will be dramatically higher and avoid this issue, because the pulls aren’t anonymous.
We have considered moving our images to our Replicated registry, and that would help for Helm installs too. No ETA on that yet though.
I see the lovely design doc in the KOTs repo – is that still roughly the plan?
If I understand this right, proxyPublicImages will solve pulling a vendor’s public images as KOTs but also the kotsadm images too for the very beginning of kubectl kots install?
On second thought, unfortunately proxyPublicImages will only handle the images in your application and not the KOTS images themselves. Thanks for catching that.
You could try to the workaround you mentioned. I don’t see why that wouldn’t work. Moving the images is a good thing to consider in the medium term though.
This is the second time we’ve hit some docker.io rate limits doing ordinary installs in a public cloud and I suspect as Docker continues to monetize, this will happen to more end users. Not super urgent for us, but on the radar.
Just an FYI, for Embedded Cluster we are doing some work to pull our own Helm charts and images through the proxy service (proxy.replicated.com). We’re also caching some things pulled through the proxy service. So one thing we could consider is patching our deployments to reference the images for KOTS and its dependencies through our proxy service. I think that would help alleviate this situation too, but let me know what you think.