title: Lost log messages when using append instead of appendRaw of LogstashMarker
body: <p>I stumbled across lost log messages in our environment.</p>
  <p>We use an ELK Stack and logback with logstash-logback-encoder in our Java applications. To do structured queries we use LogstashMarkers to append structured arguments to the logs. In the case of the lost log messages it is a string, which contains a JSON.</p>
  <p>Some of the log messages are lost completly. Not only the markers were not saved but the complete log message. They are not visible in Kibana. There are no errors in the logs of our application or the logstash server.</p>
  <p>I could create an example repository with minimal code and published it at GitHub: <a href="https://github.com/seism0saurus/logging-issue" rel="nofollow noreferrer">https://github.com/seism0saurus/logging-issue</a></p>
  <p>To run it, you need a logstash server and netcat.
  The logger in that project is configured to use a console appender, so we have a reference, what messages where really logged.
  Additionally there is one appender for a netcat server on localhost:8081 and one appender for the logstash server. Both have the same config. Only the address is different. So I'm quite sure that the message is send to the logstash server, if it is send to netcat.</p>
  <p>Here is the code of the main class:</p>
  <pre class="lang-java prettyprint-override"><code>@SpringBootApplication
  public class LoggingIssueApplication {
  
      private static final Logger LOGGER = getLogger(&quot;customLogger&quot;);
      private static final String MESSAGE = &quot;{  \&quot;ordernumber\&quot;: \&quot;Test1\&quot;,  \&quot;shippingmethod\&quot;: \&quot;homeDelivery\&quot;,  \&quot;customer\&quot;: {    \&quot;id\&quot;: \&quot;1337\&quot;,    \&quot;salutation\&quot;: \&quot;Mr\&quot;,    \&quot;name\&quot;: \&quot;seism0saurus\&quot;,    \&quot;birthdate\&quot;: \&quot;1900-05-01\&quot;,    \&quot;email\&quot;: \&quot;test@seism0saurus.de\&quot;,    \&quot;phone\&quot;: \&quot;\&quot;  },  \&quot;billingAddress\&quot;: {    \&quot;name\&quot;: \&quot;seism0saurus\&quot;,    \&quot;address\&quot;: \&quot;Teststreet 5\&quot;,    \&quot;address2\&quot;: \&quot;\&quot;,    \&quot;postcode\&quot;: \&quot;90451\&quot;,    \&quot;city\&quot;: \&quot;Nürnberg\&quot;,    \&quot;country\&quot;: \&quot;Deutschland\&quot;,    \&quot;countrycode\&quot;: \&quot;DEU\&quot;  },  \&quot;shippingAddress\&quot;: {    \&quot;name\&quot;: \&quot;seism0saurus\&quot;,    \&quot;address\&quot;: \&quot;Teststreet 5\&quot;,    \&quot;address2\&quot;: \&quot;\&quot;,    \&quot;postcode\&quot;: \&quot;90451\&quot;,    \&quot;city\&quot;: \&quot;Nürnberg\&quot;,    \&quot;country\&quot;: \&quot;Deutschland\&quot;,    \&quot;countrycode\&quot;: \&quot;DEU\&quot;  },  \&quot;totalItems\&quot;: 1,  \&quot;items\&quot;: [    {      \&quot;position\&quot;: 53,      \&quot;bundlenumber\&quot;: 53,      \&quot;itemnumber\&quot;: \&quot;900508\&quot;,      \&quot;itemdescription\&quot;: \&quot;Dinosaur Cookies\&quot;,      \&quot;quantity\&quot;: 4,      \&quot;saleType\&quot;: \&quot;KL\&quot;,      \&quot;saleItemType\&quot;: \&quot;\&quot;,      \&quot;grossAmount\&quot;: 40.0,      \&quot;discounts\&quot;: [      ]    }  ]}&quot;;
  
      public static void main(String[] args) {
          SpringApplication.run(LoggingIssueApplication.class, args);
      }
  
      @EventListener(ApplicationReadyEvent.class)
      public void testAppender() throws InterruptedException {
  
          LOGGER.warn(&quot;1 - Log without marker and message as parameter: {}&quot;, MESSAGE);
  
          LogstashMarker payloadMarker = Markers.append(&quot;payload&quot;, MESSAGE);
          LOGGER.warn(payloadMarker, &quot;2 - Log with payload marker without parameter&quot;);
          LOGGER.warn(payloadMarker, &quot;3 - Log with payload marker and message as parameter: {}&quot;, MESSAGE);
  
          LogstashMarker rawMarker = Markers.appendRaw(&quot;raw&quot;, MESSAGE);
          LOGGER.warn(rawMarker, &quot;4 - Log with raw marker without parameter&quot;);
  
          //Needed, so the loggers can finish their job before the application is shut down.
          Thread.sleep(5000);
      }
  }
  </code></pre>
  <p>And this is the logstash appender:</p>
  <pre class="lang-xml prettyprint-override"><code>  &lt;appender name=&quot;nc&quot; class=&quot;net.logstash.logback.appender.LogstashTcpSocketAppender&quot;&gt;
      &lt;destination&gt;localhost:8081&lt;/destination&gt;
      &lt;keepAliveDuration&gt;5 minutes&lt;/keepAliveDuration&gt;
      &lt;listener class=&quot;net.logstash.logback.appender.listener.FailureSummaryLoggingAppenderListener&quot;/&gt;
  
      &lt;encoder class=&quot;net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder&quot;&gt;
        &lt;providers&gt;
          &lt;mdc/&gt;
          &lt;context/&gt;
          &lt;version/&gt;
          &lt;logLevel/&gt;
          &lt;logLevelValue/&gt;
          &lt;loggerName/&gt;
  
          &lt;pattern&gt;
            &lt;pattern&gt;
              {
              &quot;serviceName&quot;: &quot;logging-issue&quot;,
              &quot;environment&quot;: &quot;dev&quot;,
              &quot;errorMessage&quot;: &quot;%ex{0}&quot;
              }
            &lt;/pattern&gt;
          &lt;/pattern&gt;
  
          &lt;threadName/&gt;
          &lt;message/&gt;
          &lt;logstashMarkers/&gt;
          &lt;arguments/&gt;
          &lt;stackTrace/&gt;
        &lt;/providers&gt;
      &lt;/encoder&gt;
    &lt;/appender&gt;
  </code></pre>
  <p>When I run the application, I get the expected 4 log messages. All 4 appear in the console and the netcat server. But only number 1 and 4 appear in kibana.
  Therefore I assume that there is a difference in handling <code>Markers.append</code> and <code>Markers.appendRaw</code>. The <code>MESSAGE</code> is a string containing JSON. But I hope, that I don't have to check the content of the string to decide which method to invoke.</p>
  <p>There are no error messages in the log of the logstash server.
  It doesn't seem to be connection errors, unknown or unrechable hosts or full buffers. I reproduced all these cases and they leave a meaningfull error message in the logs.</p>
  <p>Do you know, what happens here? Do I have to check the content of the strings I want to put into an Marker to prevent the loss of the complete log message?</p>
  <p>EDIT:
  I got more information about the problem. A logstash instance inside docker on my local machine does receive all logs and writes them to a file with the following pipeline.</p>
  <pre><code>input {
    tcp {
      port =&gt; 9000
      codec =&gt; json_lines
    }
  }
  
  output {
    file {
      path =&gt; &quot;/mounted_dir/logstash.out&quot;
      codec =&gt; line {
        format =&gt; &quot;%{message}&quot;
      }
    }
  }
  </code></pre>
  <p>EDIT 2:
  Based on <a href="https://jaxenter.de/elastic-stack-containern-docker-86374" rel="nofollow noreferrer">https://jaxenter.de/elastic-stack-containern-docker-86374</a> I build a complete ELK stack for a comparison. It doesn't loose any logmessage.
  Therefore where has to be a problem with the ELK stack, our admins build. Do you have any ideas?</p>