Following Bitnami’s recent announcement about changes to their public catalog we’ve been working with multiple software vendors to help them plan a path forward. This has resulted in a few related community guidance posts. This post is intended to be a summary of relevant helper posts and other related Q&A in regards to this impactful upstream change.
We have previously reached out by email to those software vendors that we identified as likely impacted by this upstream change based on our assessment of active release manifests. We have since partnered with several impacted vendors in the following ways: Helping determine which bitnami chart references were flagged; Helping recommend alternatives to these charts/images; Helping determine if a Bitnami compatible SecureBuild image could mitigate impact.
This is a summary of the most common paths we’ve helped vendors identify to date, as well as the considerations of each. Note that these potential workarounds highlighted are for online installs as existing air gap install release packages are immutable and unaffected (see Q&A below for more details).
For Helm CLI Installs or Embedded Cluster Installs
We recommend issuing replacement releases and demoting the previously issued bitnami-impacted releases in Vendor Portal. As described in this community post ( What happens downstream when I demote a release in Vendor Portal? (Removing a channel release) ), demoting these impacted releases will remove them from the available updates list in the Embedded Cluster Admin Console or Enterprise Portal, unless the version is already deployed by the end customer.
For KOTS existing cluster or kURL embedded cluster Installs
If a customer tries to install an old version of your application that references one of these impacted charts it will likely fail to deploy when the install assets cannot be fetched upstream.
You have a couple of potential paths forward based on what is feasible to consider with your end customers.
Workaround Option - Have your customer upgrade KOTS to the latest version while also removing any prior ‘required’ flags and demoting the impacted releases in Vendor Portal once you’ve made replacement versions available
We recommend upgrading to KOTS version 1.128.0 or newer to benefit from improved handling of releases that have their information updated post initial release. This ability to refetch release information applies only to online connected installs.
In KOTS v1.126.0 we introduced the ability for the Admin Console to refetch metadata for any releases newer than what’s deployed. This means that on “Check for Updates”, KOTS will refetch release data to look like current state in Vendor Portal.
For example:
- Your customer is currently on 1.0.0
- You’ve previously released 1.1.0 (bitnami impacted), but also made it a required release. Will likely fail to deploy after the bitnami deadline passes.
- You’ve previously released 1.2.0 as well, which is bitnami impacted and marked as a required release.
As a vendor you would now remove the required flag from 1.1.0 and 1.2.0 in Vendor Portal, so they can be skipped, and issue new non-bitnami-impacted required patch releases of 1.1.1 and 1.2.1.
You would then have your end customer update to the latest KOTS version and “Check for Updates”.
Now your end customer sees 1.1.0 and 1.2.0 as no longer required. They also now see the the option to deploy the newly required 1.1.1 and 1.2.1 releases.
As of KOTS 1.128.0 we additionally introduced the ability to hide demoted releases from the Admin Console version history page. This means that on “Check for Updates”, KOTS will refetch the release metadata, see that one or more not-yet-deployed releases have been “demoted” in upstream Vendor Portal, and therefore hide those demoted releases from the Version History UI.
For example:
- Your customer is currently on 1.0.0
- You’ve released 1.0.1 and 1.0.2, so your end customer sees 1.0.1 and 1.0.2 to deploy
- You realize that 1.0.1 has an issue and you want to remove that release from the available deployment options. You mark it as demoted in Vendor Portal
- Once the Admin Console as checked for updates, the end customer will now see only 1.0.2 as an option to deploy
Note: The above refetch functionality only applies to online installs because air gap installs leverage previously created immutable bundles. Once a bundle is uploaded, there is no need to refetch its data given its immutability.
More info can be found in this post Managing KOTS Upgrades with Upcoming Bitnami Catalog Changes
Workaround Option - Temporarily update your image path references from /bitnami to /bitnamilegacy, noting considerations below
It may not be feasible to ask your end customer to upgrade to the latest KOTS version available before August 28th (now September 29th), or you may not yet have identified replacements for your impacted images.
Ahead of re-architecting and releasing a new version of your app with replacements for impacted bitnami images, some software vendors are choosing to roll out patch releases that will keep working post August 28th (now September 29th) by changing the impacted image path from /bitnami to /bitnamilegacy, as outlined in the original bitnami announcement as a temporary workaround for this change.
This patching approach has the following considerations:
- It is a temporary workaround. Bitnami has made no commitment as to how long they intend to maintain this legacy repo
- Not all impacted charts/images appear to have a match in the /bitnamilegacy repo at this time
- We have seen a some legacy registry validation error using this path. Please be sure to reference this post on the error and what to do if you see this error: Resolving Bitnami legacy repository verification error
You will need to direct your customers to update to these newer patches and ignore previously downloaded patches that are bitnami-impacted. If none of these releases are required, the end customer can skip past them.
If your KOTS existing cluster or kURL cluster method still uses kots.io/v1beta1 or plain kubernetes manifests, you may be able to leverage kustomize to patch previously downloaded releases to point to the legacy repo vs issuing new patches. This must be done on the end customer side, and therefore will require you to provide a script to help enable this patching. More information on how to do this can be found here: Managing KOTS Upgrades with Upcoming Bitnami Catalog Changes .
Additionally if you have bitnami-impacted versions that were previously marked as “required” (cannot be skipped) then you will likely have to patch these versions using the above kustomize method if you cannot upgrade to a newer KOTS as the prior KOTS behavior would still consider these “required” releases even if you changed that later in Vendor Portal.
Other related Q&A
Question - Are previously built air gap bundles impacted?
Answer - No, previously built air gap bundles are immutable packages and should not be impacted. Only new air gap package build attempts would be impacted from the point of the upstream image change occurring.
Question - If I’m already using the Replicated Proxy registry, is that caching the upstream images?
Answer - No, the Replicated Proxy registry is strictly a proxy pull-through mechanism. There is no caching of upstream images. Therefore if the referenced path in a given release changes upstream, and now results in a failure to fetch, any newly pulled releases and new air gap build attempts will be impacted by these errors.
Question - Are any customer-facing Replicated products impacted by these Bitnami changes?
Answer - No. None of the customer-facing Replicated products (Embedded Cluster, kots, kURL) leverage upstream Bitnami charts, and are not directly impacted by this change.
If you’d like to discuss how we might help you, or if you have any additional questions or concerns, please feel free to engage your team’s Account Executive.