title: Filebeat POD getting killed due to OOM issue
body: <p>We are running workloads in Kubernetes cluster, and to ship application logs out of the cluster we are using Filebeat as daemon set, which runs on dedicated namespace.</p>
  <p>Filebeat sends the data to a Logstash cluster running behind a Load balancer.</p>
  <p>We are seeing that some of the Filebeat pods are getting restarted due to OOM, we are trying to investigate the issue, but since running in Pod, we are reaching a deadlock.</p>
  <p>Filebeat config:</p>
  <pre><code>filebeat.inputs:
  - type: container
    paths:
      - /var/lib/docker/containers/*/*.log
    processors:
      - add_kubernetes_metadata:
          in_cluster: true
          host: ${NODE_NAME}
      - drop_event.when.not.or:
          - equals.kubernetes.labels.console-log: &quot;enabled&quot;
    exclude_lines: [&quot;34mdebug&quot;]
  - type: log
    paths:
      - /var/log/node-log/messages
    include_lines:
      - 'kubelet:'
    exclude_lines:
      - 'CNI Plugin version'
    tags:
      - &quot;kubelet-logs&quot;
  - type: log
    paths:
      - /mnt/logs/*.log
    tags:
      - &quot;additional-logs&quot;
  processors:
    - drop_fields:
        fields: [&quot;agent.ephemeral_id&quot;, &quot;agent.hostname&quot;, &quot;agent.id&quot;, &quot;agent.name&quot;, &quot;agent.type&quot;, &quot;agent.version&quot;, &quot;container.id&quot;, &quot;container.image.name&quot;, &quot;container.runtime&quot;, &quot;kubernetes.container.image&quot;, &quot;kubernetes.container.name&quot;, &quot;kubernetes.namespace_labels.env&quot;, &quot;kubernetes.namespace_labels.team&quot;, &quot;kubernetes.namespace_uid&quot;, &quot;kubernetes.node.hostname&quot;, &quot;kubernetes.node.labels.beta_kubernetes_io/arch&quot;, &quot;kubernetes.node.labels.beta_kubernetes_io/instance-type&quot;, &quot;kubernetes.node.labels.beta_kubernetes_io/os&quot;, &quot;kubernetes.node.labels.failure-domain_beta_kubernetes_io/region&quot;, &quot;kubernetes.node.labels.failure-domain_beta_kubernetes_io/zone&quot;, &quot;kubernetes.node.labels.kubernetes_io/arch&quot;, &quot;kubernetes.node.labels.kubernetes_io/hostname&quot;, &quot;kubernetes.node.labels.kubernetes_io/os&quot;, &quot;kubernetes.node.labels.spotinst_io/node-lifecycle&quot;, &quot;kubernetes.node.labels.vng-group&quot;, &quot;kubernetes.node.name&quot;, &quot;kubernetes.node.uid&quot;, &quot;ecs.version&quot;, &quot;input.type&quot;, &quot;kubernetes.labels.app_kubernetes_io/managed-by&quot;, &quot;kubernetes.labels.console-log&quot;, &quot;kubernetes.pod.uid&quot;, &quot;kubernetes.labels.app_kubernetes_io/name&quot;, &quot;kubernetes.node.labels.node_kubernetes_io/instance-type&quot;, &quot;kubernetes.node.labels.topology_kubernetes_io/region&quot;, &quot;kubernetes.node.labels.topology_kubernetes_io/zone&quot;]
  
  cloud.id: ${ELASTIC_CLOUD_ID}
  cloud.auth: ${ELASTIC_CLOUD_AUTH}      
  
  fields:
    ekscluster: &quot;eks-prod&quot;
  
  output.logstash:
    hosts: [&quot;elk-logstash-internal.amazonaws.com:5043&quot;]
    timeout: 60
  </code></pre>
  <p>On checking the pod logs, we are seeing this error:</p>
  <pre><code>Mar  3 08:35:57 ip-172-20-95-255 kernel: memory: usage 204760kB, limit 204800kB, failcnt 57072
  Mar  3 08:35:57 ip-172-20-95-255 kernel: memory+swap: usage 204760kB, limit 9007199254740988kB, failcnt 0
  Mar  3 08:35:57 ip-172-20-95-255 kernel: kmem: usage 2252kB, limit 9007199254740988kB, failcnt 0
  Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB acti
  ve_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
  Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f/0575ed6294aea6f4300b85da751d0239aa4d895efba7be3bdc7e4b9cdf0ba53f: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped
  _file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
  Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f/cabb8b1f5425af10732fb04bee3437ad3c5d5c29cd7c2ebe2b5f9ddd3f18fc7c: cache:92KB rss:0KB rss_huge:0KB shmem:0KB mappe
  d_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
  Mar  3 08:35:57 ip-172-20-95-255 kernel: Memory cgroup stats for /kubepods/burstable/pod20b62bc0-c582-41ef-876a-58df4f4e073f/083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918: cache:116KB rss:202276KB rss_huge:20480KB shmem
  :0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:202408KB inactive_file:56KB active_file:0KB unevictable:0KB
  Mar  3 08:35:58 ip-172-20-95-255 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
  Mar  3 08:35:58 ip-172-20-95-255 kernel: [ 4283]     0  4283      242        1       3       2        0          -998 pause
  Mar  3 08:35:58 ip-172-20-95-255 kernel: [ 1278]     0  1278     1092      160       8       3        0           975 tini
  Mar  3 08:35:58 ip-172-20-95-255 kernel: [ 1303]     0  1303   275030    60940     168       4        0           975 filebeat
  Mar  3 08:35:58 ip-172-20-95-255 kernel: Memory cgroup out of memory: Kill process 1303 (filebeat) score 2164 or sacrifice child
  Mar  3 08:35:58 ip-172-20-95-255 kernel: Killed process 1303 (filebeat) total-vm:1100120kB, anon-rss:191432kB, file-rss:52328kB, shmem-rss:0kB
  Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.151720    2946 setters.go:77] Using node IP: &quot;172.20.95.255&quot;
  Mar  3 08:35:59 ip-172-20-95-255 dockerd: time=&quot;2022-03-03T08:35:59.249926267Z&quot; level=info msg=&quot;ignoring event&quot; container=083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918 module=libcontainerd namespace=moby topic=/tasks/de
  lete type=&quot;*events.TaskDelete&quot;
  Mar  3 08:35:59 ip-172-20-95-255 containerd: time=&quot;2022-03-03T08:35:59.251109262Z&quot; level=info msg=&quot;shim disconnected&quot; id=083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918
  Mar  3 08:35:59 ip-172-20-95-255 containerd: time=&quot;2022-03-03T08:35:59.251188224Z&quot; level=error msg=&quot;copy shim log&quot; error=&quot;read /proc/self/fd/66: file already closed&quot;
  Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.385350    2946 kubelet.go:1982] SyncLoop (PLEG): &quot;filebeat-f4h8x_filebeat(20b62bc0-c582-41ef-876a-58df4f4e073f)&quot;, event: &amp;pleg.PodLifecycleEvent{ID:&quot;20b62bc0-c582-41ef-876a-58df4f4e
  073f&quot;, Type:&quot;ContainerDied&quot;, Data:&quot;083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918&quot;}
  Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.386181    2946 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: 083febaeed20e6c5f494d84c1c53a7fb5978e618e7f42bec0fdbf1d4b3559918
  Mar  3 08:35:59 ip-172-20-95-255 kubelet: I0303 08:35:59.387106    2946 topology_manager.go:221] [topologymanager] RemoveContainer - Container ID: a81a6bbc9d15d2ea0f51c7aad5eb1fa092c97b9826c86e0701ca93914ee5ce97
  Mar  3 08:35:59 ip-172-20-95-255 containerd: time=&quot;2022-03-03T08:35:59.497230474Z&quot; level=info msg=&quot;starting signal loop&quot; namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/e956e035777aa655cb88d98efe5bb0a27208fd6ab67851
  536a3cc3407d484bb3 pid=12405
  Mar  3 08:36:00 ip-172-20-95-255 kubelet: I0303 08:36:00.413098    2946 kubelet.go:1982] SyncLoop (PLEG): &quot;filebeat-f4h8x_filebeat(20b62bc0-c582-41ef-876a-58df4f4e073f)&quot;, event: &amp;pleg.PodLifecycleEvent{ID:&quot;20b62bc0-c582-41ef-876a-58df4f4e
  073f&quot;, Type:&quot;ContainerStarted&quot;, Data:&quot;e956e035777aa655cb88d98efe5bb0a27208fd6ab67851536a3cc3407d484bb3&quot;}
  </code></pre>
  <p>Memory and CPU Limit is</p>
  <pre><code>  limits:
              cpu: 250m
              memory: 200Mi
            requests:
              cpu: 40m
              memory: 100Mi
  </code></pre>
  <p><strong>Filebeat Version</strong>  <em>7.11.2</em></p>
  <p>What can be the root cause and is there any way to investigate the issue.</p>