# Flexvolume creates deadlock and Deployment enters into Crashloopbackoff on node reboot

**URL:** <https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439>\
**Category:** Packaging an application\
**Created:** [July 29, 2020, 1:41am UTC](https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439 "2020-07-29T01:41:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![MANI\_M](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/mani_m/32/136_2.png) [@MANI\_M](https://community.replicated.com/u/MANI_M)\
**Post date:** [July 29, 2020, 1:41am UTC](https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439/1 "2020-07-29T01:41:15Z")

</div>

Hi There,  
We have a deployment with an init container where we are checking whether flexVolume is ready to be mounted. Here are the details

```auto
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: test
  namespace: replicated-namespace
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: xyz
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: xyz
        tier: backend
    spec:
      initContainers:
      - command:
        - /bin/sh
        - -c
        - df $MOUNT_PATH | grep ":6789"
        env:
        - name: MOUNT_PATH
          value: /sharedfs
        image: docker.io/replicated/replicated-operator:stable-2.46.2
        imagePullPolicy: IfNotPresent
        name: check-mount
        resources: {}
        securityContext:
          seLinuxOptions:
            type: spc_t
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sharedfs
          name: shared-mount
          readOnly: true
      containers:
      ...
      ...
      ...
      restartPolicy: Always
      volumes:
      - flexVolume:
          driver: ceph.rook.io/rook
          fsType: ceph
          options:
            clusterNamespace: rook-ceph
            fsName: rook-shared-fs
        name: shared-mount

```

Now, whenever we restart the node the Deployment goes into CrashloopBackoff. Even though Due to restart policy the deployment is recreated by k8s, it never recovers.  
It works only if we manually delete and recreate the deployment after node restart. Is there any way to get this working?

---

<div class="post-metadata">

**Author:** ![areed](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/areed/32/12_2.png) [@areed](https://community.replicated.com/u/areed)\
**Post date:** [July 29, 2020, 3:03pm UTC](https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439/2 "2020-07-29T15:03:52Z")

</div>

After reboots there is a race condition in which mounting the ceph shared filesystem usually fails. The initContainer is meant to protect against running with an unsuccessful mount. You can force delete the individual pod and allow it to be re-created by Kubernetes.

[https://help.replicated.com/docs/kubernetes/packaging-an-application/volumes/#shared-filesystem-initcontainer](https://help.replicated.com/docs/kubernetes/packaging-an-application/volumes/#shared-filesystem-initcontainer)

---

<div class="post-metadata">

**Author:** ![MANI\_M](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/mani_m/32/136_2.png) [@MANI\_M](https://community.replicated.com/u/MANI_M)\
**Post date:** [July 29, 2020, 4:43pm UTC](https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439/3 "2020-07-29T16:43:31Z")

</div>

How can this be avoided in production, Manual deletion and re-deployment may not be feasible.

---

<div class="post-metadata">

**Author:** ![areed](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/areed/32/12_2.png) [@areed](https://community.replicated.com/u/areed)\
**Post date:** [July 29, 2020, 4:53pm UTC](https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439/4 "2020-07-29T16:53:13Z")

</div>

If you’re running a clustered setup, those pods would be scheduled on other nodes when one goes down for reboot. They would not be automatically scheduled on the rebooted node (unless in a DaemonSet) so there would not be a race condition.

We’re also looking into ways to prevent or automatically fix the issue for upcoming releases.

---

<div class="post-metadata">

**Author:** ![MANI\_M](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/mani_m/32/136_2.png) [@MANI\_M](https://community.replicated.com/u/MANI_M)\
**Post date:** [July 29, 2020, 5:08pm UTC](https://community.replicated.com/t/flexvolume-creates-deadlock-and-deployment-enters-into-crashloopbackoff-on-node-reboot/439/5 "2020-07-29T17:08:38Z")

</div>

Thanks, @areed for the quick response. Yeah in dev env we are facing this issue as we are running single node cluster. In production, it should not be a problem as it is a multi-node cluster.