title: Is there a way parse log messages using rsyslog config and transform them to structured messages?
body: <p>I am trying to parse log messages and transform them to structured messages using rsyslog. Is there a way support such operation with rsyslog config? I have not yet explored the option to write custom parser or message modification plugin for this. </p>
  
  <p>I found <a href="https://www.rsyslog.com/doc/v8-stable/configuration/templates.html#property-statement" rel="nofollow noreferrer">template list properties</a> which can do some of it. Is there a way to do the following?</p>
  
  <ol>
  <li>Map 2 fields to single output name. Ex: "__ts": "2018-09-20 10:18:56.363" (first 2 fields in example below). Would not use regex here as I am looking for a solution that does not depend on value of the fields. Ex: the two fields could be two strings or some other values not just dates.</li>
  <li>Extract what is left in msg after extracting all known fields based on position. Ex: "msg": "Unregistering application nameOfAnApiHere with someOtherName with status DOWN". </li>
  <li>Is there a way to use <a href="https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html" rel="nofollow noreferrer">local variables</a> to hold the values of fields from msg and use the variables in templates? </li>
  </ol>
  
  <p><strong>Example Log message:</strong></p>
  
  <p>2018-09-20 10:18:56.363  INFO --- [Thread-68] x.y.z.key1Value Unregistering application nameOfAnApiHere with someOtherName with status DOWN</p>
  
  <p><strong>1. rsyslog config template definition</strong></p>
  
  <pre><code>template(name="structure-log-format" type="list") {
    constant(value="{")
  
    # This only extracts the first field with value 2018-09-20.
    # TODO: What is a way to map first 2 fields to map to __ts field? 
    property(outname="__ts" name="msg" field.number="1" field.delimiter="32" format="jsonf") constant(value=", ")
  
    constant(value="\"event\":[{")
      constant(value="\"payload\":{")
          property(outname="_log_" name="syslogtag" format="jsonf") constant(value=", ")
          property(outname="__loglvl" name="msg" field.number="4" field.delimiter="32" format="jsonf") constant(value=", ")
          property(outname="__thread" name="msg" field.number="7" field.delimiter="32" format="jsonf") constant(value=", ")
          property(outname="__key1" name="msg" field.number="8" field.delimiter="32" format="jsonf") constant(value=", ")
          # The following setting will include full message value starting from "2018-09-20 ... DOWN"
          # TODO: What is a way to only include message starting from "Unregistering ... DOWN"?
          property(name="msg" format="jsonf" droplastlf="on" )
      constant(value="}")
  constant(value="}]} \n")
  </code></pre>
  
  <p>}</p>
  
  <p><strong>2. Expected result:</strong> </p>
  
  <pre><code>{
     "__ts": "2018-09-20 10:18:56.363",
     "event": [
         {
            "payload": {
               "_log_": "catalina",
               "__loglvl": "INFO",
               "__thread": "Thread-68",
               "__key1": "x.y.z.key1Value",
               "msg": "Unregistering application nameOfAnApiHere with someOtherName with status DOWN"
            }
         }
       ]
  }
  </code></pre>
  
  <p><strong>3. Actual result:</strong></p>
  
  <pre><code>{
     "__ts": "2018-09-20",
     "event": [
         {
            "payload": {
               "_log_": "catalina",
               "__loglvl": "INFO",
               "__thread": "Thread-68",
               "__key1": "x.y.z.key1Value",
               "msg": "2018-09-20 10:18:56.363  INFO 2144 --- [Thread-68] x.y.z.key1Value Unregistering application nameOfAnApiHere with someOtherName with status DOWN"
            }
         }
       ]
  }
  </code></pre>
  
  <p>Thank you.</p>