title: Concatenate split log messages using fluent-concat-plugin for EKS cluster running Kubernetes 1.22
body: <p>My EKS clusters depend on Fluentd daemonsets to send log messages to ElasticSearch. Docker wraps log messages from containers, line-by-line, in JSON. It also splits log messages into 16kb chunks if they're larger than that. This causes problems when those messages are structured JSON (embedded within Docker's JSON) since they're no longer parseable.</p>
  <p>I've tried configuring the fluent-concat-plugin to identify split messages and re-assemble them before sending these to ElasticSearch. Despite my attempts, the messages either remain split, or nothing gets sent to ES.</p>
  <p>All my attempts use the following input configuration in their fluentd.conf:</p>
  <pre><code>&lt;source&gt;
    @type tail
    @id in_tail_container_logs
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    tag raw.containers.*
    read_from_head true
    &lt;parse&gt;
      @type json
      time_type string
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      keep_time_key true
    &lt;/parse&gt;
  &lt;/source&gt;
  </code></pre>
  <p>This attempt doesn't concatenate split log messages:</p>
  <pre><code>&lt;filter raw.containers.**&gt;
    @type concat
    key log
    use_partial_metadata true
    separator &quot;&quot;
  &lt;/filter&gt;
  </code></pre>
  <p>This causes nothing to appear in ES for any split message.</p>
  <pre><code>&lt;filter raw.containers.**&gt;
    @type concat
    key log
    multiline_end_regexp /\\n$/
    separator &quot;&quot;
  &lt;/filter&gt;
  </code></pre>
  <p>This blocks all processing with errors in the fluentd log indicating &quot;logtag&quot; isn't present in the JSON coming back from Docker.</p>
  <pre><code>&lt;filter raw.containers.**&gt;
    @type concat
    key log
    use_partial_cri_logtag true
    partial_cri_logtag_key logtag
    partial_cri_stream_key stream
    separator &quot;&quot;
  &lt;/filter&gt;
  </code></pre>
  <p>How should fluent-plugin-concat, or for that matter, fluentd in general, be configured to re-assemble these split log messages before further processing?</p>