Kots recycle rule to clean up old releases from Minio pod

Feature Requests
1

Currently, Kots does not have any feature where you set the amount of release version that you want to keep.

For frequently updated environments, it is a problem because sooner or later your kotsadm-minio pod will run out of disk space, making the new deployments fail.

It is a well-known issue and is documented here

This workaround is not acceptable in many cases because it involves running a script or commands to overcome the issue.

Would be nice to see a flag where you can define the number of release versions that Kots should keep. For example, I want to have only the last 20 releases. It would be like a logrotate kinda feature.

In this way, minio pod would never run out of disk space making the new deployments fail.

1 Like
2

Hi @Tiago_Farias - Thank you for bringing this request to our attention, and for submitting the request through the Vendor Portal as well. Let me discuss the potential feasibility of such a feature with the KOTS team, and then I’ll loop back with you to provide an update.

3

This recycle rule will be a nice addition.

In one of our environments we frequently run into this issue. We bump up the minio PVC size every time we see this.

# Increase kotsadm minio PVC size
kubectl patch pvc kotsadm-minio-kotsadm-minio-0 -p '{"spec": {"resources": {"requests": {"storage": "8Gi"}}}}'  
 
# Restart kotsadm minio pod
kubectl delete pod kotsadm-minio-0
4

We’ve been brainstorming a few ways we could possibly approach solving this problem, and would be interested in getting your input on the desirability of the various ideas. These are currently listed from easiest to consider implementing to harder to implement given various UI/CLI and UX considerations. Based on your post above, I’m assuming the last option is the most desirable, but I’m also interested to know if the other options would have value to your use case if it were possible to tackle those sooner.

Ideas so far:

  • A CLI based command that could be run by the end-user on demand that deletes the requested X number of oldest releases (cannot delete currently deployed or currently available release). This would remove those versions and any associated release archives (freeing up MinIO space), and they would also disappear from the UI/CLI version history after deletion.
  • A GUI based delete or archive button that the Admin Console end-user could self select any release version (except currently deployed and currently available release) to remove that version and any associated release archives from the version history.
  • Allow the vendor to put something in the app spec, or in the passed install/upgrade command, that tells KOTS to only keep the last X number of releases (cannot delete currently deployed or currently available release). Open questions - Would end-customers get confused by disappearing releases if this wasn’t end-user initiated? Would there be concern from end-customers on change-control or audibility considerations if previously available releases disappear via passed deletion commands during app updates?
5

Hi Amber,

For sure, at least for our use case, it would be much better to have it automated. So the last option is ideal.

Regarding your question, I will give my 5 cents below:

Would end-customers get confused by disappearing releases if this wasn’t end-user initiated?

A. I don’t think it would be an issue when discussing development environments that receive hundreds of weekly releases, so cleaning up very old ones is not an issue.

For production environments that receive fewer releases, you could set it to a high number, by the time the clean-up process runs, you would be running a much newer version, and deleting a two-year-old release would not be that bad.

This feature would be disabled by default anyway, so I think people would be good.

The first option would work, but it would require some additional steps to be useful, like having an automated script to run something like:

kubectl exec -it kotsadm-minio-0 -n st-master – /bin/bash -c “df -h | grep /export”

and work with this result to see if the CLI tool needs to be executed to delete old releases or not. Which is not difficult, but you need to add this additional process only for it.

6

Thank you for the input @Tiago_Farias ! Thinking about it more myself, I agree that the last option is really the most viable one. The first two require the end-user to regularly do something (or to script it as you mentioned). Given the bad state the install gets into when this MinIO pod runs out of disk space, preventative deletion is safer that relying on an end-user side process.

We’re continuing to talk about this idea internally. I cannot give you a clear timeline yet on when we might prioritize this work, but we’ll update you as that prioritization status changes.

8

Would like to bump this feature request.

We have staging and development clusters that get dozens of version updates per day, and it exhausted the storage space in Minio.

Increasing the storage space is a stopgap solution, as this is bound to happen again. KOTS needs to rotate the data it stores, purging data from older versions and keeping only data for newer ones

9

@miguelvr Thank you for the additional info on your specific use case, and the input that this feature would also be useful for you.