title: Fluent Bit 1.8+ and MULTILINE_PARSER
body: <p>My goal is to collect logs from Java (Spring Boot) applications running on Bare Kubernetes.
  These logs are then translated into ES and visualized in Kibana.</p>
  <p>For these purposes I deployed Fleunt Bit 1.8.9 via Kubernetes 1.22. Since I use Containerd instead for Docker, then my Fluent Bit configuration is as follow (Please note that I have only specified one log-file):</p>
  <pre><code>  fluent-bit.conf: |
      [SERVICE]
          Flush         1
          Log_Level     info
          Daemon        off
          Parsers_File  parsers.conf
          HTTP_Server   On
          HTTP_Listen   0.0.0.0
          HTTP_Port     2020
  
      @INCLUDE input-kubernetes.conf
      @INCLUDE filter-kubernetes.conf
      @INCLUDE output-elasticsearch.conf
  
    input-kubernetes.conf: |
      [INPUT]
          Name              tail
          Tag               kube.*
          Path              /var/log/containers/*.log
          Read_from_head    true
          Parser            cri
  
    filter-kubernetes.conf: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Kube_URL            https://kubernetes.default.svc:443
          Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
          Kube_Tag_Prefix     kube.var.log.containers.
          Merge_Log           On
          Merge_Log_Key       log_processed
          K8S-Logging.Parser  On
          K8S-Logging.Exclude Off
  
    output-elasticsearch.conf: |
      [OUTPUT]
          Name     es
          Match    kube.*
          Host     ${FLUENT_ELASTICSEARCH_HOST}
          Port     ${FLUENT_ELASTICSEARCH_PORT}
          Index    kube-code_index
          Type     kube-code_type
  
    parsers.conf: |
      [PARSER]
          Name cri
          Format regex
          Regex ^(?&lt;time&gt;[^ ]+) (?&lt;stream&gt;stdout|stderr) (?&lt;logtag&gt;[^ ]*) (?&lt;message&gt;.*)$
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  </code></pre>
  <p>With this configuration in Kibana, the Java stack trace messages are displayed unstructured:
  <a href="https://i.stack.imgur.com/9dhBM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9dhBM.png" alt="enter image description here" /></a></p>
  <p>But I need the Java stack trace to be structured like in the screenshot below:
  <a href="https://i.stack.imgur.com/ZztwH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZztwH.png" alt="enter image description here" /></a></p>
  <p>I tried a configuration like this:</p>
  <pre><code>  input-kubernetes.conf: |
      [INPUT]
          Name              tail
          Tag               kube.*
          Path              /var/log/containers/*.log
          Read_from_head    true
          Multiline.parser  cri, multiline-regex-cri
  </code></pre>
  <p>And:</p>
  <pre><code>parsers.conf: |
      [PARSER]
          # http://rubular.com/r/tjUt3Awgg4
          Name cri
          Format regex
          Regex ^(?&lt;time&gt;[^ ]+) (?&lt;stream&gt;stdout|stderr) (?&lt;logtag&gt;[^ ]*) (?&lt;message&gt;.*)$
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  
      [MULTILINE_PARSER]
          name          multiline-regex-cri
          type          regex
          flush_timeout 1000
          #
          # Regex rules for multiline parsing
          # ---------------------------------
          #
          # configuration hints:
          #
          #  - first state always has the name: start_state
          #  - every field in the rule must be inside double quotes
          #
          # rules |   state name  | regex pattern                  | next state
          # ------|---------------|--------------------------------------------
          rule      &quot;start_state&quot;   &quot;/(\D+ \d+ \d+\:\d+\:\d+)(.*)/&quot;  &quot;cont&quot;
          rule      &quot;cont&quot;          &quot;/^\s+at.*/&quot;                     &quot;cont&quot;
  </code></pre>
  <p>But all the same, the logs are not structured:
  <a href="https://i.stack.imgur.com/YUK8c.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YUK8c.png" alt="enter image description here" /></a></p>
  <p>Please help me structure the logs.</p>