How to fix failed to create fsnotify watcher Error When Running support-bundle

How do I?
, ,
1

If you encounter errors when creating a support bundle or viewing logs in a Kubernetes cluster, such as:

  • kubectl logs output: failed to create fsnotify watcher: too many open files
  • pod log from support bundle shows: failed to create fsnotify watcher: too many open files at the end of log file

The error occurs because the system has reached the limit for file watchers, which is set by the fs.inotify.max_user_watches and fs.inotify.max_user_instances parameters on the host system (k8s cluster nodes).

This issue is related to the inotify limits on the Kubernetes nodes themselves.

The solution needs to be applied on the host system (k8s cluster nodes) because containers use the same kernel as their host.

  1. Check the current limits first
cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances
  1. Increase value of max_user_watches and max_user_instances
sudo sysctl -w fs.inotify.max_user_watches= 8192 (default for kURL)
sudo sysctl -w fs.inotify.max_user_watches= 65536 (default for kURL)
  1. Verifying the Changes
After updating the limit, try running:

kubectl logs <pod-name>

to confirm that the error no longer appears.
  1. Make the Change Permanent
To persist the new limit across reboots, add the following line to /etc/sysctl.conf:
echo 'fs.inotify.max_user_instances = 8192' | sudo tee -a /etc/sysctl.conf
echo 'fs.inotify.max_user_watches = 65536' | sudo tee -a /etc/sysctl.conf

Then apply the change with:
sudo sysctl -p