Upgrade guide: Velero 1.16 to 1.17+ on kURL/KOTS with LVP snapshots

How do I?
, ,
1

Customer upgrade guide: Velero 1.16 to 1.17+ on kURL/KOTS with LVP snapshots

Scope: This guide applies only to Replicated KOTS/kURL installations where
Velero snapshots are configured to use the Local Volume Provider (LVP):
NFS, Host Path, or Internal Storage. If you are not on LVP, this guide
does not apply.

Status: This is a manual workaround for an upstream Velero change. Velero
1.17+ no longer supports the Local Volume Provider (LVP) by design; kURL/KOTS
customers must migrate snapshots to a kopia-compatible backend before upgrading.
The mkdir /udmrepo error described in the troubleshooting section is a symptom
of the in-place upgrade path; the migration steps below avoid it by deleting the Velero
pods before the upgrade. Contact your Replicated support contact if you need help.

What is changing and why

Velero 1.17 and later replace the restic uploader with kopia and no longer
support the Local Volume Provider (LVP). This is an upstream Velero design choice,
not a Replicated bug. If your current KOTS snapshot destination is implemented using
LVP (NFS, Host Path, or Internal Storage), the kURL installer will refuse to upgrade
Velero until the destination is migrated to a kopia-compatible backend.

Important: This guide is only for clusters whose BackupStorageLocation
is LVP-backed. Some KOTS installs already use file-system Minio for NFS/Host
Path/Internal Storage destinations, in which case the BackupStorageLocation provider
is aws and the s3Url points at an in-cluster kotsadm-fs-minio service. This
guide does not apply to those installs. See the verification step below before
proceeding.

The migration path is to keep the same physical storage (e.g., the same NFS
export or host path) but access it through an in-cluster Minio that presents
an S3-compatible interface to Velero. KOTS calls this the file-system Minio
path. Your backup bytes still live on the same NFS share or local disk; only the
API Velero uses to reach them changes.

Important: The old restic backups that were written by Velero 1.16 cannot
be restored by Velero 1.17+. Treat this upgrade as a cut-over: keep the old
snapshot data until you are confident in the new kopia backups, but do not expect
to restore from the old restic repository after the upgrade.

Prerequisites

  • A kURL/KOTS cluster currently on Velero 1.16 with snapshots configured to NFS,
    Host Path, or Internal Storage.
  • kubectl access to the cluster (admin-level, because you will delete Velero
    controller resources).
  • The KOTS Admin Console password and the new kURL installer URL from your vendor
    (the one that upgrades both KOTS and Velero to 1.17+).
  • A maintenance window. Snapshot restores are not possible during the cut-over,
    and old snapshots become unusable after the BackupStorageLocation is deleted.

Recommended pre-upgrade checklist

  1. Notify stakeholders that snapshot restores from before the upgrade will not
    be available after the cut-over.

  2. Verify you are actually on LVP. This guide is only for clusters whose
    BackupStorageLocation is backed by the Local Volume Provider. Check the BSL:

    kubectl describe backupstoragelocation -n velero default

    If you see Provider: replicated.com/nfs, replicated.com/hostpath,
    or replicated.com/local, this guide applies to you.

    If you see any other provider (for example, aws with an s3Url pointing at
    kotsadm-fs-minio), this guide does not apply. Stop here and use the
    standard KOTS/Velero upgrade guidance instead.

  3. Take a final Velero backup on Velero 1.16 if you want the most recent
    possible recovery point. This is the last backup that uses the old restic
    repository.

  4. Record the current snapshot destination (server, path, credentials) so you
    can re-enter it in the KOTS Admin Console after the upgrade.

  5. Verify the NFS share / host path is still accessible and has enough free
    space for new kopia backups.

Upgrade steps

Step 1 — Delete the old LVP snapshot configuration

Run these commands from a node with kubectl access to the cluster. These remove
the LVP-backed BackupStorageLocation, the old restic BackupRepositories, the
internal Velero PVC, and the Velero controller + node-agent workloads.

kubectl delete backupstoragelocation -n velero default
kubectl delete backuprepository -n velero --all
kubectl delete pvc -n velero velero-internal-snapshots  # ignore "not found" if it does not exist
kubectl delete deployment -n velero velero
kubectl delete daemonset -n velero node-agent

Confirm nothing is left that would block the installer:

kubectl get backupstoragelocation -n velero
kubectl get backuprepository -n velero
kubectl get pvc -n velero
kubectl get deployment -n velero velero
kubectl get daemonset -n velero node-agent

All of the above should return either No resources found or Error from server (NotFound).

Why this is necessary: The kURL Velero 1.17 installer blocks if it detects
an existing LVP BackupStorageLocation. Deleting it clears the gate. Deleting the
Velero Deployment and node-agent DaemonSet forces the installer to recreate them
with the correct kopia pod spec.

Step 2 — Run the kURL installer that upgrades KOTS and Velero

This step upgrades KOTS to at least version v1.131.6

The installer for this step needs to have:

  • KOTS version 1.313.6 or later
  • Velero v1.17.x

Run the installer URL provided. This is typically executed on the cluster’s primary node.

curl -sSL https://kurl.sh/<your-1.17-installer-id> | sudo bash

Wait for the installer to complete. Because there is no LVP BackupStorageLocation
left, the Velero 1.17 addon should install successfully.

Step 3 — Reconfigure the snapshot destination in the KOTS Admin Console

After the installer finishes, open the KOTS Admin Console and go to:

Snapshots → Settings

Re-enter the same destination you used before:

  • Destination type: NFS (or Host Path / Internal Storage, as before)
  • Server / Path: the same NFS server and export path, or the same host path

KOTS will deploy an in-cluster kotsadm-fs-minio instance and point Velero at it as
an S3-compatible backend. The actual backup bytes will still be stored on the same
NFS share or host path.

You can also do this from the command line. For an NFS destination:

kubectl kots velero configure-nfs \
  --namespace velero \
  --nfs-server <NFS_SERVER_IP> \
  --nfs-path <NFS_EXPORT_PATH>

For Host Path or Internal Storage, use the equivalent kubectl kots velero configure-*
command or the KOTS Admin Console.

Step 4 — Verify the new kopia repositories are Ready

From a node with kubectl access:

kubectl get backupstoragelocation -n velero default
kubectl get backuprepository -n velero
kubectl describe backuprepository -n velero default-default-kopia

You should see:

  • BackupStorageLocation/default with Provider: aws and an s3Url pointing at
    kotsadm-fs-minio.
  • BackupRepository resources whose REPOSITORY TYPE is kopia and whose
    Phase is Ready.

If the BackupRepository type is still restic, wait a few minutes and re-run
kubectl get backuprepository. If it does not switch to kopia, re-run the
snapshot destination configuration in the Admin Console or contact support.

Step 5 — Take a test backup and confirm it lands on storage

Trigger a snapshot from the KOTS Admin Console, or from the command line:

velero backup create test-after-upgrade --include-namespaces <your-app-namespace>

Then verify on the storage side:

  • For NFS: log into the NFS server and check the export path. The directory structure
    has changed: under the original export path you will see two new hidden directories
    (names starting with .) and a new velero directory. The kopia data is inside the
    velero directory, alongside any old restic data.
  • For Host Path: check the configured host path on the node; the same new directory
    layout applies.

Example for NFS:

ls -la /srv/nfs/velero

You should see two hidden directories (names starting with .) and a velero
directory. The exact names of the hidden directories are generated by Minio and do not
need to be inspected directly. The kopia data is inside the velero directory.

Step 6 — Decide what to do with the old restic data

After you have confirmed that new kopia backups are working and you no longer need
to roll back to Velero 1.16, you may delete the old restic directory from the NFS
share or host path to reclaim space. The old restic backups are not restorable by
Velero 1.17+.

Keep the old data until you are confident in the new backups and have tested at
least one restore from a new kopia snapshot.

Troubleshooting

The kURL installer still fails with “Local Volume Provider not supported”

Run:

kubectl get backupstoragelocation -n velero
kubectl describe backupstoragelocation -n velero default

If any BackupStorageLocation exists with Provider: replicated.com/nfs,
replicated.com/hostpath, or replicated.com/local, delete it before re-running
the installer. If the provider is anything else, this guide does not apply.

Backups fail with mkdir /udmrepo: permission denied

This error was observed when the node-agent DaemonSet or velero Deployment was
upgraded in place instead of being deleted and recreated by the kURL installer. The
cleanest fix is to delete both workloads and re-run the kURL installer that includes
Velero 1.17+:

kubectl delete deployment -n velero velero
kubectl delete daemonset -n velero node-agent
# Re-run the kURL installer URL

In the validated clean-upgrade path, the /udmrepo directory does not appear as a
separate volume mount, yet backups succeed. Only patch the volume if you are unable to
re-run the installer and you still see the permission-denied error. Contact support for
help if needed.

BackupRepositories stay restic after the upgrade

If kubectl get backuprepository -n velero still shows restic repositories after
the upgrade, delete them and re-run the snapshot destination configuration in the
KOTS Admin Console:

kubectl delete backuprepository -n velero --all

Then re-save Snapshots → Settings in the Admin Console. New kopia repositories
should be created automatically.

Restores from old snapshots fail

This is expected for any Velero 1.16 → 1.17+ upgrade, not only LVP migrations.
Velero 1.17+ cannot read restic repositories created by Velero 1.16. The only
recovery is to restore from a kopia snapshot taken after the upgrade.

Limitations and caveats

  • Old snapshots are not restorable after the upgrade. Plan for a maintenance
    window and take any final backups you need before starting.
  • Minio on NFS is a single-instance filesystem Minio. This is the supported
    KOTS pattern for the file-system Minio path, but it is not the same as a
    multi-node Minio deployment. Customers with very high backup throughput should
    discuss sizing with support.
  • This guide was validated on kURL/KOTS. It does not apply to other
    deployment types.
  • Always test a restore from a new kopia snapshot before deleting the old restic
    data.