title: Uncomprehensible out of Memory Error with Logstash
body: <p>I have a <strong>Logstash 7.6.2 docker</strong> that stops running because of memory leak. After each pipeline execution, it looks like Logstash doesn't release memory.
  <strong>What should I do to identify the source of the problem?
  How can I solve it?</strong></p>
  
  <p>Any help is welcom ^^.</p>
  
  <p>Here is the error I see in the logs. These are just the 5 first lines of the Traceback. I uploaded the rest in a file in my github <a href="https://github.com/yviouswilliamous/Logstash-error/blob/master/logstash_traceback.txt" rel="nofollow noreferrer">there</a>.</p>
  
  <pre><code>logstash    | [2020-04-08T18:15:42,960][INFO ][logstash.outputs.file    ][rawweb] Closing file /output/web_data.json
   logstash    | [2020-04-08T18:15:43,353][ERROR][org.logstash.Logstash    ] java.lang.OutOfMemoryError: Java heap space
   logstash    | [2020-04-08T18:15:43,367][ERROR][org.logstash.execution.WorkerLoop][rawclient] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
   logstash    | org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `pop' for nil:NilClass
   logstash    |   at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.awesome_print_minus_1_dot_7_dot_0.lib.awesome_print.inspector.awesome(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/inspector.rb:117) ~[?:?]
  </code></pre>
  
  <p>Here the <strong>docker-compose.yml</strong> I used to configure my Logstash Docker</p>
  
  <pre><code>version: '2.4'    
  services:
        logstash:
          image: docker.elastic.co/logstash/logstash:7.6.2
          container_name: logstash
          environment:
            LS_JAVA_OPTS: "-Xmx7g -Xms4g"
            REQUEST_FREQUENCY: 600 #seconds
          volumes:
            - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
            - ./logstash/pipelines.yml:/usr/share/logstash/config/pipelines.yml
            - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
            - ./logstash/tests:/testscripts:ro
            - /root/logstash_output/:/output/
          ports:
            - "9600:9600"
          mem_limit: 7000M
          mem_reservation: 100M
  </code></pre>
  
  <p>My <strong>pipelines.yml</strong> file</p>
  
  <pre><code>- pipeline.id: rawclient
    path.config: "/usr/share/logstash/pipeline/logclient.conf"
    pipeline.batch.size: 10000000
  
   - pipeline.id: rawweb
     path.config: "/usr/share/logstash/pipeline/logweb.conf"
     pipeline.batch.size: 10000000
  </code></pre>
  
  <p>One of my <strong>.conf</strong> files. Basically, it executes a <strong>.sh</strong> script containing a curl request. The result of this request is the input of the pipeline. Treatments are made. Then results are stored in file. The two pipelines do the same, the only difference is the curl request that is made.</p>
  
  <pre><code>input {
    exec {
      command =&gt; "bash /testscripts/logclient_1.sh"
      codec =&gt; "json"
      interval =&gt; "600"
    }
  }
  
  filter {
      mutate {
          rename =&gt; ["connection/start_time", "start_time" ]
          rename =&gt; ["connection/end_time", "end_time" ]
          rename =&gt; ["connection/duration", "duration" ]
          rename =&gt; ["connection/destination_ip_address", "destination_ip_address" ]
          rename =&gt; ["connection/status", "status" ]
          rename =&gt; ["device/last_ip_address", "last_ip_address" ]
          rename =&gt; ["user/sid", "sid" ]
          # rename =&gt; ["binary/application_category", "application_category" ]
          rename =&gt; ["binary/application_name", "application_name" ]
          rename =&gt; ["binary/executable_name", "executable_name" ]
          remove_field =&gt; ["@timestamp"]
          remove_field =&gt; ["@version"]
          add_field =&gt; { "connection_type" =&gt; "client" }
      }
  }
  output {
    file {
     path =&gt; "/output/client_data.json"
     codec =&gt; "json"
   }
    stdout {
     codec =&gt; rubydebug
   }
  }
  </code></pre>
  
  <p>My <strong>logstash.yml</strong> file</p>
  
  <pre><code>http.host: "0.0.0.0"
  xpack.monitoring.enabled: false
  Thanks for all the help :slightly_smiling_face:
  </code></pre>