kURL: How can I delete rook-ceph when you have kots application installed

How do I?
1

If you are experiencing some cluster issues with kotsadm, kotsadm-postgres-0 and prometheus in crashing and failed in volume mount. You may need to re-install you rook-ceph. To delete rook-ceph completely, you cannot just run kubectl delete namespace rook-ceph, which will stuck in terminating status and cannot be re-installed.

The right steps are as follows:

  1. export all the pod, deployments, statefulsets, configmaps and secrets
  • kubectl get <type> -o yaml > default-<type>.yaml. This is useful both for recreating the config AND for “everything is on fire let’s just go back to what the application was before”
  1. scale down ekco, kotsadm and prometheus
  • kubectl scale deployment -n kurl ekc-operator --replicas=0
  • kubectl scale statefulset kotsadm-postgres --replicas=0
  • kubectl patch prometheus -n monitoring k8s --type='json' --patch '[{"op": "replace", "path": "/spec/replicas", value: 0}]'
  1. delete all PVCs using by rook
  • kubectl get pvc -A
  1. delete all PVs using by rook
  2. run the deletion procedure that kurl would:
  • cat tasks.sh | bash -s remove-rook-ceph
  1. run the SAME kurl installer as they installed with, INCLUDING KURL VERSION, with force-reapply-addons
  • cat install.sh | bash -s force-reapply-addons
  1. upload license to kotsadm console
  2. recreate settings (this is what the exported files are for!)
  3. kubectl apply
  4. repeat 8+9 until you actually get it right