title: Getting docker logs to graylog2 AND locally
body: <p>I've been working on this problem for a couple days now and have had no luck finding answers.</p>
  
  <p>The setup is that I have several hosts running groups of docker containers, and I need to have local copies of all container logs AND on a centralized graylog2 server.</p>
  
  <p>The graylog2 server is up and running and has inputs for GELF on port 9000 and syslog UDP on 514.</p>
  
  <ol>
  <li><p>Originally, containers were logging to JSON files on the docker hosts. I then changed the log-driver in docker to use GELF and send to the graylog2 server. This worked fine, however it turns out we need local copies of the logs as well. The best part about this is that the logs on graylog had the fields filled out nicely, including things like "container_name" and "source" (which was the hostname of the docker host).</p></li>
  <li><p>So I reverted to making containers log to JSON and then just sent the JSON logs to graylog2, however when I did that, graylog2 just showed one long string for the message which included the source, container_name, container_id, and so on but not in any of the corresponding fields.</p></li>
  <li><p>Then I tried changing the log driver to syslog and having rsyslog or syslog-ng send a copy of the logs to graylog2, but that created entries on graylog2 that was missing a bunch of data, this time the message only contained the actual message, there was no info at all about the container_name or container_id.</p></li>
  <li><p>Then I tried changing the log driver back to GELF and having syslog-ng listen on each docker host and docker was sending the logs to the host it was sitting on. This has a similar result as solution #2 above, so I see a very long "message" field that contains host, container_name, container_id, timestamp, etc etc, but graylog2 doesn't seem to want to populate the corresponding fields. Additionally, in this method all logs have the log source set to the IP address of the docker host even though I use hostnames for everything.</p></li>
  <li><p>I tried installing graylog-collector-sidecar but that didn't seem to work at all.</p></li>
  </ol>
  
  <p>Does anyone have any suggestions on where to go from here? I'm thinking option #3 would be good if I could somehow include at least the container_name to show up (maybe with a tag). The other solution that I think would be good is if I can get #4 to actually populate the fields, meaning instead of having this:</p>
  
  <pre><code>source: $IP
  message: $IP {"version","1.1","host":"$HOSTNAME","message":"$MESSAGE","container_name":"$CONTAINER_NAME","container_id":"$CONTAINER_ID",etc etc}
  </code></pre>
  
  <p>it should display this:</p>
  
  <pre><code>source: $HOSTNAME
  container_name: $CONTAINER_NAME
  container_id: $CONTAINER_ID
  message: $MESSAGE
  </code></pre>
  
  <p>Anyone know how I can get graylog2 (or syslog-ng) to format the log data so that it looks like the lower example? Note that when I was sending log data directly from docker to graylog2 using the GELF log-driver the data did appear in that format and it was great, we just need to also keep a copy of the logs locally.</p>
  
  <p>Thanks in advance for any input!
  -Martin</p>