KOTS: MinIO pod ran out of storage, how can I free up space?

Supporting your customers
1

If the MinIO pod that is deployed as part of the KOTS admin console runs out of storage we can free up some space by scaling down the kotsadm-minio statefulset and deleting the kotsadm-minio pvc.

Please note that this will clear the version history from the admin console, only proceed if this is acceptable for your instance.

  1. First you will want to save your application’s config file. This can be found in the admin console under the “view files” tab, it is the: upstream/userdata/config.yaml file.
    The config will be used again later when it’s time to reinstall the application.

  2. Remove the application reference by running: kubectl kots remove <app-slug> -n <namespace> -f

  3. Scale down MinIO: kubectl scale statefulset kotsadm-minio --replicas=0 -n <namespace>

  4. Delete the MinIO PVC: kubectl delete pvc kotsadm-minio-kotsadm-minio-0 -n <namespace>

  5. Scale up MinIO: kubectl scale statefulset kotsadm-minio --replicas=1 -n <namespace>

  6. Now the application can be reinstalled, this can be done via the cli or ui.

Installing via the CLI:

Reinstalling the application via the CLI can be done with the following command:
kubectl kots install <app-slug> --license-file <license.yaml> --namespace <namespace> --config-values <config-values.yaml>
This is where we can provide the application’s configuration that was saved in step 1.

For more information on kots install see:

Installing via the UI:

If the application is being reinstalled via the ui it is recommended to delete the current kotsadm pod first.

  • Find the current kotsadm pod name: kubectl get pods -n <namespace>
  • Delete the pod:
    kubectl delete pod <kotsadm-pod-name> -n <namespace>
  • Command to access admin console (if needed): kubectl kots admin-console -n <namespace> and continue the installation through the admin console.
  • The application’s config values will have to be entered manually via the ui.
1 Like
2

If KOTS is using the minio deployment from the minio namespace and not kotsadm-minio, we recommend the following:

  1. Download the existing config values - Sharing a ConfigValues File | Replicated Docs
  2. Exec into kotsadm pod
  3. export S3_HOST=$(echo $S3_ENDPOINT | awk -F/ ‘{print $3}’)
  4. s3cmd --access_key=$S3_ACCESS_KEY_ID --secret_key=$S3_SECRET_ACCESS_KEY --host=$S3_HOST --no-ssl --host-bucket=$S3_BUCKET_NAME.$S3_HOST del s3://$S3_BUCKET_NAME --recursive --force
  5. Scale down kotsadm and kotsadm-rqlite to 0
  6. Delete kotsadm-rqlite pvc
  7. Scale kotsadm and kotsadm-rqlite back up to 1
  8. Re-install app using the downloaded config values