title: Elasticsearch: Grok-pipeline not working (Not applying to logs)
body: <p>i created a pipeline: &quot;ExtractOwaspErrorFields&quot; with a grok processor which extracts fields from the message field. It is working perfectly when testing it in Kibana -&gt; Stack Management -&gt; Ingest Pipelines -&gt; Test Pipeline.
  I use a real Log-Document by providing _id and _index of a Document. The processor extracts fields from the message fields just as expected. When i add the processor to my filebeat.yml no documents are shown in 'Discovery' at all:</p>
  <pre class="lang-yaml prettyprint-override"><code>output.elasticsearch:
    hosts: [&quot;elasticsearch:9200&quot;]
    username: xxx
    password: xxx
    pipeline: ExtractOwaspErrorFields
  </code></pre>
  <p>My pipeline is configure as follows:</p>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;ExtractOwaspErrorFields&quot; : {
      &quot;processors&quot; : [
        {
          &quot;grok&quot; : {
            &quot;field&quot; : &quot;message&quot;,
            &quot;patterns&quot; : [
              &quot;%{OWASP_ERRORLOG}&quot;
            ],
            &quot;pattern_definitions&quot; : {
              &quot;OWASP_ERRORLOG&quot; : &quot;\\[%{HTTPDERROR_DATE:timestamp}\\] \\[:%{LOGLEVEL:loglevel}\\] \\[pid %{POSINT:pid}:tid %{NUMBER:tid}\\]?( \\[client %{IPORHOST:client}:%{POSINT:clientport}\\]) \\[client %{IPORHOST:client_ip}\\] ModSecurity: ?(%{APACHE_ERROR_MESSAGE:error}) \\[file \&quot;%{PATH:matching_rule_file}\&quot;\\] \\[line \&quot;%{POSINT:matching_rule_line}\&quot;\\] \\[id \&quot;%{NUMBER:matching_rule_id}\&quot;\\] \\[msg \&quot;%{DATA:owasp_message_string}\&quot;\\] \\[data \&quot;%{DATA:owasp_message_data}\&quot;\\] \\[severity \&quot;%{WORD:owasp_severity}\&quot;\\] \\[ver \&quot;%{DATA:owasp_version}\&quot;\\] %{GREEDYDATA:tags} \\[hostname \&quot;%{HOSTNAME:hostname}\&quot;\\] \\[uri \&quot;%{URIPATHPARAM:uri}\&quot;\\] \\[unique_id \&quot;%{DATA:unique_id}\&quot;\\]&quot;,
              &quot;APACHE_ERROR_MESSAGE&quot; : &quot;( .+?(?= \\[%{WORD} \&quot;))&quot;
            },
            &quot;if&quot; : &quot;ctx?.docker.container.labels.com_docker_stack_namespace == 'modsecurity'&quot;,
            &quot;ignore_failure&quot; : true
          }
        }
      ]
    }
  }
  </code></pre>
  <p>My sample document for testing the pipeline is:</p>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;docs&quot;:
    [
      {
        &quot;_id&quot;: &quot;OzAKyoIBILrgz4V8VcpG&quot;,
        &quot;_index&quot;: &quot;filebeat-7.17.5-2022.08.02-000001&quot;,
        &quot;_source&quot;: {
          &quot;docker&quot;: {
            &quot;container&quot;: {
              &quot;labels&quot;: {
                &quot;com_docker_stack_namespace&quot;: &quot;modsecurity&quot;
              }
            }
          },
          &quot;ecs&quot;: {
            &quot;version&quot;: &quot;1.12.0&quot;
          },
          &quot;stream&quot;: &quot;stderr&quot;,
          &quot;message&quot;: &quot;[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \&quot;(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\&quot; at ARGS:test. [file \&quot;/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\&quot;] [line \&quot;71\&quot;] [id \&quot;930110\&quot;] [msg \&quot;Path Traversal Attack (/../)\&quot;] [data \&quot;Matched Data: ../ found within ARGS:test: ../\&quot;] [severity \&quot;CRITICAL\&quot;] [ver \&quot;OWASP_CRS/3.3.2\&quot;] [tag \&quot;modsecurity\&quot;] [tag \&quot;application-multi\&quot;] [tag \&quot;language-multi\&quot;] [tag \&quot;platform-multi\&quot;] [tag \&quot;attack-lfi\&quot;] [tag \&quot;paranoia-level/1\&quot;] [tag \&quot;OWASP_CRS\&quot;] [tag \&quot;capec/1000/255/153/126\&quot;] [hostname \&quot;modsecurity\&quot;] [uri \&quot;/\&quot;] [unique_id \&quot;YwSeR-gNfcPQLAl5gStNfAAAAQE\&quot;]&quot;
        }
      }
    ]
  }
  </code></pre>
  <p>Result of Testing the pipeline:</p>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;docs&quot;: [
      {
        &quot;doc&quot;: {
          &quot;_index&quot;: &quot;filebeat-7.17.5-2022.08.02-000001&quot;,
          &quot;_type&quot;: &quot;_doc&quot;,
          &quot;_id&quot;: &quot;OzAKyoIBILrgz4V8VcpG&quot;,
          &quot;_source&quot;: {
            &quot;owasp_severity&quot;: &quot;CRITICAL&quot;,
            &quot;owasp_message_string&quot;: &quot;Path Traversal Attack (/../)&quot;,
            &quot;pid&quot;: &quot;226&quot;,
            &quot;error&quot;: &quot; Warning. Pattern match \&quot;(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\&quot; at ARGS:test.&quot;,
            &quot;tid&quot;: &quot;139758264993536&quot;,
            &quot;clientport&quot;: &quot;41062&quot;,
            &quot;docker&quot;: {
              &quot;container&quot;: {
                &quot;labels&quot;: {
                  &quot;com_docker_stack_namespace&quot;: &quot;modsecurity&quot;
                }
              }
            },
            &quot;owasp_version&quot;: &quot;OWASP_CRS/3.3.2&quot;,
            &quot;hostname&quot;: &quot;modsecurity&quot;,
            &quot;ecs&quot;: {
              &quot;version&quot;: &quot;1.12.0&quot;
            },
            &quot;stream&quot;: &quot;stderr&quot;,
            &quot;client&quot;: &quot;10.0.1.48&quot;,
            &quot;client_ip&quot;: &quot;10.0.1.48&quot;,
            &quot;timestamp&quot;: &quot;Tue Aug 23 11:30:47.675452 2022&quot;,
            &quot;unique_id&quot;: &quot;YwSeR-gNfcPQLAl5gStNfAAAAQE&quot;,
            &quot;matching_rule_id&quot;: &quot;930110&quot;,
            &quot;message&quot;: &quot;[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \&quot;(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\&quot; at ARGS:test. [file \&quot;/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\&quot;] [line \&quot;71\&quot;] [id \&quot;930110\&quot;] [msg \&quot;Path Traversal Attack (/../)\&quot;] [data \&quot;Matched Data: ../ found within ARGS:test: ../\&quot;] [severity \&quot;CRITICAL\&quot;] [ver \&quot;OWASP_CRS/3.3.2\&quot;] [tag \&quot;modsecurity\&quot;] [tag \&quot;application-multi\&quot;] [tag \&quot;language-multi\&quot;] [tag \&quot;platform-multi\&quot;] [tag \&quot;attack-lfi\&quot;] [tag \&quot;paranoia-level/1\&quot;] [tag \&quot;OWASP_CRS\&quot;] [tag \&quot;capec/1000/255/153/126\&quot;] [hostname \&quot;waf\&quot;] [uri \&quot;/\&quot;] [unique_id \&quot;YwSeR-gNfcPQLAl5gStNfAAAAQE\&quot;]&quot;,
            &quot;uri&quot;: &quot;/&quot;,
            &quot;owasp_message_data&quot;: &quot;Matched Data: ../ found within ARGS:test: ../&quot;,
            &quot;tags&quot;: &quot;[tag \&quot;modsecurity\&quot;] [tag \&quot;application-multi\&quot;] [tag \&quot;language-multi\&quot;] [tag \&quot;platform-multi\&quot;] [tag \&quot;attack-lfi\&quot;] [tag \&quot;paranoia-level/1\&quot;] [tag \&quot;OWASP_CRS\&quot;] [tag \&quot;capec/1000/255/153/126\&quot;]&quot;,
            &quot;loglevel&quot;: &quot;error&quot;,
            &quot;matching_rule_file&quot;: &quot;/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf&quot;,
            &quot;matching_rule_line&quot;: &quot;71&quot;
          },
          &quot;_ingest&quot;: {
            &quot;timestamp&quot;: &quot;2022-08-23T12:08:05.5017157Z&quot;
          }
        }
      }
    ]
  }
  </code></pre>
  <p>Can somebody tell me what is wrong with my pipeline or what i missed to do. Why is it working in &quot;Test Pipeline&quot; but not when i use them for real logs in filebeat.yml
  Thank you very much</p>