# Log Collection for Crashed and Terminated Pods

**URL:** <https://community.replicated.com/t/log-collection-for-crashed-and-terminated-pods/1103>\
**Category:** Supporting your customers\
**Tags:** support\
**Created:** [January 26, 2023, 9:17am UTC](https://community.replicated.com/t/log-collection-for-crashed-and-terminated-pods/1103 "2023-01-26T09:17:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thomas\_Corvazier](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/thomas_corvazier/32/618_2.png) [@Thomas\_Corvazier](https://community.replicated.com/u/Thomas_Corvazier)\
**Post date:** [January 26, 2023, 9:17am UTC](https://community.replicated.com/t/log-collection-for-crashed-and-terminated-pods/1103/1 "2023-01-26T09:17:34Z")

</div>

Hi all,

Currently, our support-bundle only includes logs from running and recently terminated pods.

However, it is important to also have access to logs from crashed or terminated pods in order to troubleshoot and resolve issues effectively.

I am considering using Collectd or Fluentd with local file output.

I would appreciate any other recommendations you may have for log collection solutions.

Thanks,  
Thomas

---

<div class="post-metadata">

**Author:** ![diamon\_w](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/diamon_w/32/150_2.png) [@diamon\_w](https://community.replicated.com/u/diamon_w)\
**Post date:** [January 26, 2023, 9:30pm UTC](https://community.replicated.com/t/log-collection-for-crashed-and-terminated-pods/1103/2 "2023-01-26T21:30:14Z")

</div>

Hello @Thomas_Corvazier 👋

Are you referring to Pods which are no longer visible with `kubectl get pods`? If so, you’re correct that you’d want to aggregate the logs somewhere so they persist as Pods come and go.

We do have a [suggestion](https://github.com/replicatedhq/troubleshoot/issues/613) on the Troubleshoot Github repo for a collector that could retrieve logs from an aggregator like ElasticSearch/Logstash, but today I believe something like Fluentd with local file output and our [CopyFromHost](https://troubleshoot.sh/docs/collect/copy-from-host/) collector would be your best path forward.

---

<div class="post-metadata">

**Author:** ![Thomas\_Corvazier](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/thomas_corvazier/32/618_2.png) [@Thomas\_Corvazier](https://community.replicated.com/u/Thomas_Corvazier)\
**Post date:** [January 27, 2023, 4:57pm UTC](https://community.replicated.com/t/log-collection-for-crashed-and-terminated-pods/1103/3 "2023-01-27T16:57:21Z")

</div>

Thanks for your response Diamon  
Yes, correct, I meant pods deleted by the cluster and not showing in `kubectl get pods`.  
After some investigation, it seems I will need a log collector that could work with many kubernetes distributions and using fluentd seems to make a lot of assumptions on how log files are stored in the cluster (stored in /var/log/containers? JSON or plain text?) so maybe the best would be to use a k8s API based log collector instead?

---

<div class="post-metadata">

**Author:** ![diamon\_w](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.replicated.com/diamon_w/32/150_2.png) [@diamon\_w](https://community.replicated.com/u/diamon_w)\
**Post date:** [March 2, 2023, 4:34pm UTC](https://community.replicated.com/t/log-collection-for-crashed-and-terminated-pods/1103/4 "2023-03-02T16:34:40Z")

</div>

Sorry @Thomas_Corvazier for the extremely late reply here.

> so maybe the best would be to use a k8s API based log collector instead?

I think I’d need some context on what you mean by a `k8s API based log collector`. Are you referring to what we have available today - [Pod Logs - Troubleshoot Docs - Troubleshoot Docs](https://troubleshoot.sh/docs/collect/logs/)? As you have already discovered, this will only collect logs from the containers that are still running or visible with `kubectl get pods`.

To capture logs from pods that no longer exist you would need some sort of log aggregation. I think your idea about fluentd would be the easy way to go about this. I imagine that each node would have the logs available at `/some/file/path/to/fluentd/logs` as you mentioned, and then you could scoop them up with [Copy Files and Directories from Hosts - Troubleshoot Docs - Troubleshoot Docs](https://troubleshoot.sh/docs/collect/copy-from-host/) from each node in the cluster. Does that make sense?