# Managing nodes when the previous Rook version is in use might leave Ceph in an unhealthy state where mon pods are not rescheduled

**URL:** https://community.replicated.com/t/managing-nodes-when-the-previous-rook-version-is-in-use-might-leave-ceph-in-an-unhealthy-state-where-mon-pods-are-not-rescheduled/1099
**Category:** Supporting your customers
**Tags:** kurl, rook
**Created:** [January 24, 2023, 9:21am UTC](https://community.replicated.com/t/managing-nodes-when-the-previous-rook-version-is-in-use-might-leave-ceph-in-an-unhealthy-state-where-mon-pods-are-not-rescheduled/1099 "2023-01-24T09:21:41Z")
**Posts on this page:** 1
**Showing post:** 1

<div class="post-metadata">

### Author: ![Camila\_Macedo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/camila_macedo/32/603_2.png) [@Camila\_Macedo](https://community.replicated.com/u/Camila_Macedo)
#### Post date: [January 24, 2023, 9:21am UTC](https://community.replicated.com/t/managing-nodes-when-the-previous-rook-version-is-in-use-might-leave-ceph-in-an-unhealthy-state-where-mon-pods-are-not-rescheduled/1099/1 "2023-01-24T09:21:41Z")

</div>

After following the [steps](https://kurl.sh/docs/install-with-kurl/managing-nodes#remove-nodes-with-ecko) to remove a node in hosts and then add a new one you might face the following issue scenario when the host installation is using an old Rook version (i.e. Ceph version `14.2.0` installed with Rook `1.0.4`).

> NOTE: This specific scenario can be faced if you are using Rook `1.0.4` but it is not reproducible by using the latest versions. You can check that it is recommended to upgrade the Rook version in use as described in [kURL docs](https://kurl.sh/docs/install-with-kurl/managing-nodes#rook-ceph-cluster-prerequisites).

**If you checked that the Ceph is unhealthy with the out of quorum message:**

By running the command `kubectl -n rook-ceph exec deployment.apps/rook-ceph-operator -- ceph status`, i.e.:

 ![unhelth-ceph-check](https://canada1.discourse-cdn.com/flex030/uploads/replicated/original/1X/2aa2a6721837e8bd26a83fbb49db8b20e838dba5.jpeg)

**If you checked that the mon Rook pod for the purged node is still in usage:**

By running `kubectl -n rook-ceph get pod -l app=rook-ceph-mon`, i.e.:

 ![mon-pod-peding](https://canada1.discourse-cdn.com/flex030/uploads/replicated/original/1X/6222c495e51ee0b3b1d8e216f9181e9d700cc67e.png)

Therefore, in order to workaround this scenario and make Ceph came back to a health state you might able to achieve it by performing manual steps such as described in [Mon is never rescheduled · Issue #2262 · rook/rook · GitHub](https://github.com/rook/rook/issues/2262#issuecomment-460898915) by Rook maintainers. Please, ensure that you check the following guidance and steps.

## How to sort out the Ceph warning state to upgrade

In this case, you can check that the info about the purged node still in the mapping spec by running `kubectl -n rook-ceph describe configmaps rook-ceph-mon-endpoints | grep <node-name> `. Therefore, you can:

1. Stop the Rook Operator by running: `kubectl -n rook-ceph scale --replicas=0 deployment.apps/rook-ceph-operator`
2. Edit the configmap rook-ceph-mon-endpoints to **(carefully)** remove the purged node info from the mapping with the command `kubectl -n rook-ceph edit configmaps rook-ceph-mon-endpoints` Following an example:

In this example, we purged the node `example-ubuntu-2204-node-c` but we can still checking it in the mapping:

 ![Screenshot 2023-01-26 at 08.55.42](https://canada1.discourse-cdn.com/flex030/uploads/replicated/original/1X/9193763da763e05f59774d5d27646c8df63037a6.jpeg)

In this way, to workaround the scenario and try to make the Ceph came back to a health state we will remove `"c": ["Name": "example-ubuntu-2204. node-c". "Hostname" : "example-ubuntu-2204-node-c" "Address"': "10.154.0.21"},`

1. Delete the mon pod in Pending state. You can found its name by running ` kubectl -n rook-ceph get pod -l app=rook-ceph-mon` and you can remove by running `kubectl -n rook-ceph delete pod <mon-pod-name-in-pending-state>`
2. Then, rescale the Rook Operator by running: `kubectl -n rook-ceph scale --replicas=1 deployment.apps/rook-ceph-operator`

## After performing the above fix, please ensure that the Ceph comes back to a health state

1. Check that all Rook Mon pod(s) are running with `kubectl -n rook-ceph get pod -l app=rook-ceph-mon`, i.e.:

 ![Screenshot 2023-01-24 at 08.04.59](https://canada1.discourse-cdn.com/flex030/uploads/replicated/original/1X/4574b863360bbba035bf5f44fa918f5f31e3b6f0.png)

1. Verify the Ceph status with `kubectl -n rook-ceph exec deployment.apps/rook-ceph-operator -- ceph status` to ensure that it is in a health state, i.e.

 ![Screenshot 2023-01-24 at 08.06.14](https://canada1.discourse-cdn.com/flex030/uploads/replicated/original/1X/74b0e717cc0e3fb659efc21a59ce10aa8a71ef0b.jpeg)

---

_[View the full topic](https://community.replicated.com/t/managing-nodes-when-the-previous-rook-version-is-in-use-might-leave-ceph-in-an-unhealthy-state-where-mon-pods-are-not-rescheduled/1099)._