Best Practices for Executing Upgrades of KOTS/kURL infrastructure

Supporting your customers
1

Had a great question come in today that isn’t super well documented anywhere today.

We have several instances in the field across embedded and existing clusters. The app version is easy to keep up to date, but we’re wondering what are the best practices for upgrading the KOTS App Manager version for these installs in the field, as well as upgrading Kubernetes and other underlying kURL components.

Tactical Execution

  1. Test all upgrades in a representative environment in your infrastructure. That means, install the exact version of your app, KOTS App Manager, (and the kURL stack if applicable) and then test the upgrade path for upgrading App Manager or kURL or both. This will mean building an understanding of the state of each customer whom you’re looking to upgrade. You can review this in the Customer Reporting section of https://vendore.replicated.com/customers.
  2. It is highly recommended that you enable and test Replicated’s Snapshot and Restore capabilities so that you can take a pre-upgrade snapshot in case anything goes wrong
  3. Minimize incremental upgrade scope. While it can be tempting to try to upgrade k8s, kots, and your network layer all in one go, it is much better to do 3 smaller upgrades, testing in isolation, than to try to upgrade all of these things at once. This is especially true for upgrades involving Kubernetes, network plugins, or storage providers.

Performing Upgrades

KOTS app Manager is released weekly, and today the way to upgrade it in an Existing Cluster to use the kots CLI.

  1. Install the latest version of the CLI with curl https://kots.io/install | bash (or download a release from the releases page Releases · replicatedhq/kots · GitHub)
  2. Run kubectl kots admin-console upgrade --namespace $NAMESPACE using the namespace where the kotsadm workloads are deployed.

Upgrading an Embedded Cluster

To upgrade an online (non-airgapped) kURL cluster, end customers should be instructed to re-run the idempotent initial install command to upgrade kURL packages and containers.

NOTE If you have pinned the kURL version using Versioned kURL Releases, you will need to update your pinned version and promote the new installer to customer facing channel(s) before issuing the above instructions. Alternatively or in parallel, you can hard-code the kURL release version in the installer URL before providing it to your end customers, noting specifically:

If an installer version is specified in both the yaml and the URL, the URL version will be used.

NOTE 2 In both cases, the customer should ensure they use the same flags as were used for the initial installation.

Online example:

curl -fsSL https://k8s.kurl.sh/wordpress-enterprise | sudo bash [-s FLAGS]

To upgrade an airgapped kURL cluster, a new airgap bundle should be generated and installed.

curl -fSL -o wordpress-enterprise.tar.gz https://k8s.kurl.sh/bundle/wordpress-enterprise.tar.gz
# ... scp or sneakernet wordpress-enterprise.tar.gz to airgapped machine, then
tar xvf wordpress-enterprise.tar.gz
sudo bash ./install.sh airgap [FLAGS]

Specific (non-example) commands for each of these operations can be queried using the replicated vendor CLI with a properly configured REPLICATED_APP and REPLICATED_API_TOKEN. Note that Stable is used below but the channel(s) you use for production customers may be different.

replicated channel inspect Stable
2

What the best way to notify customers they have to upgrade cluster? Our cluster installer manifest bundled with application (beta functionality) which is very convenient. We make a release of our application and made changes in installer, lets say change version of k8s. Customer will get notification about app upgrade in admin console, but won’t get notification that he needs perform upgrade of cluster. Do we need notify them directly via e-mail?

3

Right now we don’t have a great way of alerting customers of when they have a cluster spec upgrade available so we rely on notification via the vendor to let them know when they have to update their cluster. That could be email or something else right now. I think it makes sense to add in functionality to let customers know when their clusters need to be updated though, we will discuss this and figure out what this feature might look like. I will reach out on slack for more details.