title: Bringing in single and multi-line App log records to ELK (some contain JSON objects)
body: <p>I'm trying to take log records from a custom (node.js) application that will be putting data into elastic search and then processed by Kibana. My environment is Ubuntu with ELK (Elasticsearch, Logstash and Kibana) and the log generating Application is in Node.JS</p>
  
  <p>I'm already processing the standard system log files, like syslog, nginx. The ELK environment and the Apps are on different servers</p>
  
  <p>Since this is a log file that is created by our App it contains entries that have various patterns. But each entry does start with a common header 
  [example - <em>2015-03-17T11:26:27.285Z (INFO|dev3)</em> ) Creating interaction document...] 
  which is 
  [<em>date+time stamp (msg-level|system-ID) Some message text</em> ]</p>
  
  <p>Frequently this is the entire log entry. But sometimes it can be followed by a JSON object. Depending on the message being generated it may have a different JSON Object. If a JSON object is included(starting on the next line) the line will end with "..." (minus the quotes) but not all lines that end that way have a JSON object following.</p>
  
  <p>As a first step I'd line to bring in the entire multi-line JSON object as part of the message. Right now I'm using the syslog filter and each line comes in as a separate message. Then my ultimate goal would be to parse the JSON objects and have the contained fields stored separately so that Kibana can cleanly filter on their individual values.</p>
  
  <p>From what I've seen so far there are two ways this can be done.</p>
  
  <ul>
  <li>Create logstash filters to parse and then pass to Elasticesearch</li>
  <li>Have the Application write the information directly to the elasticsearch database. Maybe with something like log4j. </li>
  </ul>
  
  <p>My first question is which method would be the most flexible in the long run ? Creating a multi-line filter and importing the JSON object as a single message is probably the quickest. But if writing directly to elastic search makes it easier to bring in different JSON objects and have the individual fields available to the filters that would probably be my longer term goal.</p>
  
  <p>I am including some dummy sample log data below to show what I am looking to process</p>
  
  <p>Thanks</p>
  
  <pre><code>2015-03-17T11:26:27.285Z (INFO|dev3) Creating interaction document...
  { req_url: '/nL4sWsw',
    company_cookie: '68d1dc4a32ed3bfd22c96a6e60a132924e5d8fa8',
    browsing_cookie: '68d1dc4a32ed3bfd22c96a6e60a132924e5d8fa8',
    visit_count: 1,
    campaign_id: 52d6ab20bbc1e6ac0500032f,
    switchboard_id: 54888c6ffc4ac2cb18a3b8c6,
    content_key: '2d0515120561b7be80c936027f6dce71b41a0391',
    http_header: 
     { 'x-host': 'subdomain.company.org',
       'x-ip': '111.222.333.444',
       host: 'logic9',
       connection: 'close',
       'user-agent': 'Mozilla/5.0 (compatible; ext-monitor - premium monitoring service; http://www.ext-monitor.com)' },
    timestamp: Tue Mar 17 2015 06:26:27 GMT-0500 (CDT),
    url: 'https://cdn.company.org/2d0515120561b7be80c936027f6dce71b41a0391/',
    type7id: 'nL4sWsw',
    pid: undefined,
    onramp_type: 'type7',
    http_user_agent: 'Other',
    http_browser: 'Other' }
  2015-03-17T11:26:27.285Z (INFO|dev3) Inserting interactions data...
  { 'statistics.total_interactions': 1,
    'statistics.day_of_week.tuesday': 1,
    'statistics.onramp_type.type7': 1,
    'statistics.hour_of_day.11': 1,
    'statistics.operating_systems.other': 1,
    'statistics.browser_types.other': 1 }
  2015-03-17T11:26:27.286Z (INFO|dev3) Updating campaign 52d6ab20bbc1e6ac0500032f with stats {"statistics.total_interactions":1,"statistics.day_of_week.tuesday":1,"statistics.onramp_type.type7":1,"statistics.hour_of_day.11":1,"statistics.operating_systems.other":1,"statistics.browser_types.other":1} ...
  2015-03-17T11:26:27.286Z (INFO|dev3) Redirecting to https://cdn.company.org/2d0515120561b7be80c936027f6dce71b41a0391/ ...
  2015-03-17T11:26:27.286Z (INFO|dev3) Campaign statistics recorded successfully
  2015-03-17T11:26:27.287Z (INFO|dev3) GET /zVoxiPV
  2015-03-17T11:26:27.287Z (INFO|dev3) GET /vumkm3A
  2015-03-17T11:26:27.287Z (INFO|dev3) Starting response for type7v1 ...
  2015-03-17T11:26:27.287Z (INFO|dev3) Header: {"x-host":"subdomain.company.org","x-ip":"111.222.333.444","host":"logic9","connection":"close","user-agent":"Mozilla/5.0 (compatible; ext-monitor - premium monitoring service; http://www.ext-monitor.com)"}
  2015-03-17T11:26:27.287Z (INFO|dev3) Params: {"tid":"zVoxiPV"}
  2015-03-17T11:26:27.287Z (INFO|dev3) Sending taIdentity cookie: f79b8ceca66f99608fb1291ab51d65b08fa3138f ...
  2015-03-17T11:26:27.287Z (INFO|dev3) Sending taBrowse cookie: f79b8ceca66f99608fb1291ab51d65b08fa3138f ...
  2015-03-17T11:26:27.287Z (INFO|dev3) Sending new cookie: 96ec5414d0b847790f58a1feee2399d282cf7907 with visit count 1 ...
  2015-03-17T11:26:27.288Z (INFO|dev3) Finding in switchboard {"active":true,"campaign.start_at":{"$lte":"2015-03-17T11:26:27.287Z"},"campaign.end_at":{"$gte":"2015-03-17T11:26:27.287Z"},"type7id":"zVoxiPV"}
  2015-03-17T11:26:27.288Z (INFO|dev3) Starting response for type7v1 ...
  2015-03-17T11:26:27.288Z (INFO|dev3) Header: {"x-host":"subdomain.company.org","x-ip":"111.222.333.444","host":"logic9","connection":"close","user-agent":"Mozilla/5.0 (compatible; ext-monitor - premium monitoring service; http://www.ext-monitor.com)"}
  2015-03-17T11:26:27.288Z (INFO|dev3) Params: {"tid":"vumkm3A"}
  2015-03-17T11:26:27.288Z (INFO|dev3) Sending taIdentity cookie: adec72a656ef7999d101edc7e1e9cf901e1e56c9 ...
  2015-03-17T11:26:27.288Z (INFO|dev3) Sending taBrowse cookie: adec72a656ef7999d101edc7e1e9cf901e1e56c9 ...
  2015-03-17T11:26:27.288Z (INFO|dev3) Sending new cookie: 0c1354b30bf261595bf24a14c2e90ecac64545ed with visit count 1 ...
  2015-03-17T11:26:27.288Z (INFO|dev3) Finding in switchboard {"active":true,"campaign.start_at":{"$lte":"2015-03-17T11:26:27.288Z"},"campaign.end_at":{"$gte":"2015-03-17T11:26:27.288Z"},"type7id":"vumkm3A"}
  2015-03-17T11:26:27.289Z (INFO|dev3) Finding in matching set [object Object]
  2015-03-17T11:26:27.289Z (INFO|dev3) Switchboard item {"_id":"5488a7ea60c5508693bebba7","content_provider":"redirect","content":{"_id":"54b8954eca0ca5eb87cb4fef","name":"Content for Switchboard 5488a7ea60c5508693bebba7","key":"ad354806eadd0f90ef55b1ab96a8c84272401186"},"type":"redirect","campaign":{"end_at":"2018-12-11T00:00:00.000Z","start_at":"2008-12-11T00:00:00.000Z","_id":"52a9dd9bfb9c94150600032f"}}
  2015-03-17T11:26:27.289Z (INFO|dev3) No url for redirect, going local...
  2015-03-17T11:26:27.289Z (INFO|dev3) url: https://cdn.company.org/ad354806eadd0f90ef55b1ab96a8c84272401186/
  2015-03-17T11:26:27.289Z (INFO|dev3) Sending redirect to https://cdn.company.org/ad354806eadd0f90ef55b1ab96a8c84272401186/ ...
  2015-03-17T11:26:27.289Z (INFO|dev3) Creating interaction document...
  { req_url: '/zVoxiPV',
    company_cookie: 'f79b8ceca66f99608fb1291ab51d65b08fa3138f',
    browsing_cookie: 'f79b8ceca66f99608fb1291ab51d65b08fa3138f',
    visit_count: 1,
    campaign_id: 52a9dd9bfb9c94150600032f,
    switchboard_id: 5488a7ea60c5508693bebba7,
    content_key: 'ad354806eadd0f90ef55b1ab96a8c84272401186',
    http_header: 
     { 'x-host': 'subdomain.company.org',
       'x-ip': '111.222.333.444',
       host: 'logic9',
       connection: 'close',
       'user-agent': 'Mozilla/5.0 (compatible; ext-monitor - premium monitoring service; http://www.ext-monitor.com)' },
    timestamp: Tue Mar 17 2015 06:26:27 GMT-0500 (CDT),
    url: 'https://cdn.company.org/ad354806eadd0f90ef55b1ab96a8c84272401186/',
    type7id: 'zVoxiPV',
    pid: undefined,
    onramp_type: 'type7',
    http_user_agent: 'Other',
    http_browser: 'Other' }
  </code></pre>