title: Fluentd logs not sent to Elasticsearch - pattern not match
body: <p>I can't get log messages to be processed by Fluentd and sent to Elasticsearch. I'm tailing the container of my service, it will pick up the log, but it can't parse it, it always fails with the error <strong>pattern not match</strong>. I understand that something is wrong with my parsing setup, but I can't see what.</p>
  <p>The service writes to Stdout with Serilog, using the ElasticsearchJsonFormatter. My understanding is that it will write valid json to the console. This appears to be happening if I view the logs of the running container. When I view the Fluentd logs, it looks as if it has all been escaped.</p>
  <p>If I view the logs of the service pod I can see the message, if then view the logs of the fluentd pod I can see the pattern not match error. Both of these are included below.</p>
  <p>Any help or pointers will be greatly appreciated, as I've been stuck on this for days now.</p>
  <h2>Serilog Setup</h2>
  <pre><code>var loggerConfig = new LoggerConfiguration()
                   .WriteTo.Console(new ElasticsearchJsonFormatter())
                   .MinimumLevel.Information()
                   .MinimumLevel.Override(&quot;Microsoft&quot;, LogEventLevel.Warning)
                   .MinimumLevel.Override(&quot;System&quot;, LogEventLevel.Warning);
  </code></pre>
  <h2>Fluentd Config</h2>
  <pre><code>apiVersion: v1
  kind: ConfigMap
  metadata:
    name: fluentd-config
  data:
    fluent.conf: |
      &lt;source&gt;
        @type tail
        path /var/log/containers/my-service-*.log
        pos_file /var/log/app.log.pos
        tag kubernetes.*
        read_from_head true
        &lt;parse&gt;
          @type json
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        &lt;/parse&gt;
      &lt;/source&gt;
  
      &lt;match **&gt;
        @type elasticsearch
        host &quot;#{ENV['FLUENT_ELASTICSEARCH_HOST']}&quot;
        port &quot;#{ENV['FLUENT_ELASTICSEARCH_PORT']}&quot;
        user &quot;#{ENV['FLUENT_ELASTICSEARCH_USER']}&quot;
        password &quot;#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}&quot;
        index_name fluentd
        type_name fluentd
      &lt;/match&gt;
  
  </code></pre>
  <h2>Example log message</h2>
  <p>This is what I can see if I view the logs for the running container. In my case this is a Kubernetes pod.</p>
  <pre><code>{
      &quot;@timestamp&quot;: &quot;2021-12-29T13:23:28.2692128+00:00&quot;,
      &quot;level&quot;: &quot;Information&quot;,
      &quot;messageTemplate&quot;: &quot;Get Record - User Claim Contracts: {contracts} All Claims: {allclaims}&quot;,
      &quot;message&quot;: &quot;Get Record - User Claim Contracts: [1, 2, 3, 4] All Claims: \&quot;Application.Claims\&quot;&quot;,
      &quot;fields&quot;: {
          &quot;contracts&quot;: [
              1,
              2,
              3,
              4
          ],
          &quot;allclaims&quot;: &quot;Application.Claims&quot;,
          &quot;ddsource&quot;: &quot;application&quot;,
          &quot;RecordId&quot;: null,
          &quot;UserId&quot;: null,
          &quot;TransactionId&quot;: null,
          &quot;env&quot;: &quot;UAT&quot;,
      }
  }
  </code></pre>
  <h2>Fluentd pattern not match</h2>
  <p>This is what I see when I view the logs for the fluentd container. Again, this is a pod.</p>
  <pre><code>2021-12-29 13:37:48 +0000 [warn]: #0 pattern not match: &quot;2021-12-29T13:37:47.852294372Z stdout F {\&quot;@timestamp\&quot;:\&quot;2021-12-29T13:37:47.8518242+00:00\&quot;,\&quot;level\&quot;:\&quot;Information\&quot;,\&quot;messageTemplate\&quot;:\&quot;Get Record - User Claim Contracts: {contracts} All Claims: {allclaims}\&quot;,\&quot;message\&quot;:\&quot;Get Record - User Claim Contracts: [1, 2, 3, 4] All Claims: \\\&quot;Application.Claims\\\&quot;\&quot;,\&quot;fields\&quot;:{\&quot;contracts\&quot;:[1,2,3,4],\&quot;allclaims\&quot;:\&quot;\&quot;,\&quot;ddsource\&quot;:\&quot;\&quot;,\&quot;RecordId\&quot;:null,\&quot;UserId\&quot;:null,\&quot;TransactionId\&quot;:null,\&quot;env\&quot;:\&quot;UAT\&quot;,\&quot;\&quot;:\&quot;\&quot;,\&quot;\&quot;:\&quot;\&quot;}}&quot;
  </code></pre>