title: Change an event's field using the aggregate filter in Logstash
body: <p>For some days now I am struggling with a problem.</p>
  
  <p>I have logs that contain the actions of the users and are of the format:    </p>
  
  <p>For example: </p>
  
  <p>07/04/2016 00:05:57.821 reqId:1 Request Begin message body</p>
  
  <p>07/04/2016 00:06:57.821 reqId:1 Request End message body</p>
  
  <p>07/04/2016 00:15:57.821 reqId:2 Request Begin message body</p>
  
  <p>07/04/2016 01:35:57.821 reqId:3 Request Begin message body</p>
  
  <p>The request ID which is unique for each action.</p>
  
  <p>Each action begins with "Request Begin" and ends with "Request End". However, there are some actions that never end (maybe due to a Java error), that is, there will not be a matching "Request End" for a specific request ID.</p>
  
  <p>Using Logstash I would like to correlate the events that contain "Request Begin" with their matching "Request End" and then, using Kibana I want to retrieve all actions that do NOT contain a matching "Request End" (that is the unfinished ones.</p>
  
  <p>I came across the aggregate filter. But I can not figure out how to configure it to correlate the events.</p>
  
  <p>A snippet of my logstash.conf is the following:</p>
  
  <pre><code>if [request-status] == "Request Begin"{
              aggregate {
                  task_id =&gt; "%{request-ID}"
                  code =&gt; "map['request-complete']; event['request-complete']=0"
                  map_action =&gt; "create_or_update"
                  }
      }
      if [request-status] == "Request End"{
              aggregate {
                  task_id =&gt; "%{request-ID}"
                  code =&gt; "event ['request-complete'] = 1"
                  map_action =&gt; "update"
                  timeout =&gt; 600
                  end_of_task =&gt; true
                        }
      }
  </code></pre>
  
  <p>The output is :</p>
  
  <pre><code>Logstash startup completed
  {
               "message" =&gt; "07/04/2016 00:05:57.821 reqId:1 Request Begin message body",
              "@version" =&gt; "1",
            "@timestamp" =&gt; "2016-04-22T11:44:53.102Z",
        "real-timestamp" =&gt; "07/04/2016 00:05:57.821",
            "request-ID" =&gt; "1",
        "request-status" =&gt; "Request Begin",
          "message-body" =&gt; "message body",
      "request-complete" =&gt; 0
  }
  {
               "message" =&gt; "07/04/2016 00:06:57.821 reqId:1 Request End message body",
              "@version" =&gt; "1",
            "@timestamp" =&gt; "2016-04-22T11:44:53.400Z",
        "real-timestamp" =&gt; "07/04/2016 00:06:57.821",
            "request-ID" =&gt; "1",
        "request-status" =&gt; "Request End",
          "message-body" =&gt; "message body",
      "request-complete" =&gt; 1
  }
  {
               "message" =&gt; "07/04/2016 00:15:57.821 reqId:2 Request Begin message body",
              "@version" =&gt; "1",
            "@timestamp" =&gt; "2016-04-22T11:44:53.400Z",
        "real-timestamp" =&gt; "07/04/2016 00:15:57.821",
            "request-ID" =&gt; "2",
        "request-status" =&gt; "Request Begin",
          "message-body" =&gt; "message body",
      "request-complete" =&gt; 0
  }
  {
               "message" =&gt; "07/04/2016 01:35:57.821 reqId:3 Request Begin message body",
              "@version" =&gt; "1",
            "@timestamp" =&gt; "2016-04-22T11:44:53.401Z",
        "real-timestamp" =&gt; "07/04/2016 01:35:57.821",
            "request-ID" =&gt; "3",
        "request-status" =&gt; "Request Begin",
          "message-body" =&gt; "message body",
      "request-complete" =&gt; 0
  }
  </code></pre>
  
  <p><strong>My question:</strong></p>
  
  <p>Is there any way I can change the field "request-complete" of a "Request Begin" action to match the one of its corresponding "Request End"?</p>
  
  <p>For example I would like to change the "request-complete" of the event with "request-ID => 1" and "request-status => Request Begin", from "request-complete => 0" to "request-complete => 1" (which is the same value as the event with "request-ID => 1" and "request-status => Request End").</p>
  
  <p>This way I can search ,in Kibana, the events that have "request-status =Request Begin" AND "request-complete =1" (which will be all events without a matching closing event).</p>
  
  <p>Thanks in advance for any help</p>