# Removing nodes from Kubernetes clusters

**URL:** <https://community.replicated.com/t/removing-nodes-from-kubernetes-clusters/371>\
**Category:** Supporting your customers\
**Created:** [October 30, 2019, 7:03pm UTC](https://community.replicated.com/t/removing-nodes-from-kubernetes-clusters/371 "2019-10-30T19:03:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![areed](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/areed/32/12_2.png) [@areed](https://community.replicated.com/u/areed)\
**Post date:** [October 30, 2019, 7:03pm UTC](https://community.replicated.com/t/removing-nodes-from-kubernetes-clusters/371/1 "2019-10-30T19:03:36Z")

</div>

In an HA Kubernetes cluster the REK operator will automatically purge failed nodes that have been unreachable for more than an hour. For master nodes this includes the following steps:

1. Delete the Deployment resource for the OSD from the `rook-ceph` namespace
2. Exec into the Rook operator pod and run the command `ceph osd purge <id>`
3. Delete the Node resource
4. Remove the node from the `CephCluster` resource named `rook-ceph` in the `rook-ceph` namespace unless storage is managed automatically with `useAllNodes: true`
5. (Masters only) Connect to the etcd cluster and remove the peer
6. (Masters only) Remove the apiEndpoint for the node from the `kubeadm-config` ConfigMap in the `kube-system` namespace

All of these steps can be performed manually if needed. For removing etcd peers, exec into one of the remaining etcd pods in the `kube-system` namespace. You can use the `etcdctl` CLI there with the certificates mounted in `/etc/kubernetes/pki/etcd`:

```auto
$ cd /etc/kubernetes/pki/etcd
$ etcdctl --endpoints=https://127.0.0.1:2379 --cert-file=healthcheck-client.crt --key-file=healthcheck-client.key --ca-file=ca.crt member list

a1316b56d7099abf: name=node-k7d4 peerURLs=https://10.128.0.124:2380 clientURLs=https://10.128.0.124:2379 isLeader=false
ab67f9f870c32907: name=node-wbf1 peerURLs=https://10.128.0.125:2380 clientURLs=https://10.128.0.125:2379 isLeader=false
d9228c5ac755a5c6: name=node-hrrm peerURLs=https://10.128.0.123:2380 clientURLs=https://10.128.0.123:2379 isLeader=true

$ etcdctl --endpoints=https://127.0.0.1:2379 --cert-file=healthcheck-client.crt --key-file=healthcheck-client.key --ca-file=ca.crt member remove a1316b56d7099ab

```

---

<div class="post-metadata">

**Author:** ![dmitriy](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/dmitriy/32/16_2.png) [@dmitriy](https://community.replicated.com/u/dmitriy)\
**Post date:** [September 10, 2020, 4:14pm UTC](https://community.replicated.com/t/removing-nodes-from-kubernetes-clusters/371/2 "2020-09-10T16:14:45Z")

</div>

Newer versions of `etcdctl` use different command line arguments

```auto
--cert=healthcheck-client.crt --key=healthcheck-client.key --cacert=ca.crt

```