anchor.diff:  ...
   }
   ----
  +
  +== Engine Extensions
  +
  +Engine extensions are listeners that can operate on Data or Events received before the engine process them.
  +
  +Extensions can implement a variety of use cases where transformation or filtering of incoming Data or Events might be necessary.
  +
  +Extensions are executed in a pipeline ordered by registration time.
  +
  +Extensions must implement a DataExtension or EventExtension interface and be registered through the `ExtensionsService`.
  +
  +=== DataExtension interface
  +
  +[source,java]
  +----
  +public interface DataExtension {
  +
  +    /**
  +     * The extension processes the supplied Data and returns Data to be forwarded, if any.
  +     *
  +     * @param data The Data to be processed by the extension.
  +     * @return The set of Data to be forwarded to the next extension, or core engine if this is the final extension.
  +     */
  +    TreeSet<Data> processData(TreeSet<Data> data);
  +
  +}
  +----
  +
  +=== EventExtension interface
  +
  +[source,java]
  +----
  +public interface EventExtension {
  +
  +    /**
  +     * The extension processes the supplied Events and returns Events to be forwarded, if any.
  +     *
  +     * @param events The Events to be processed by the extension.
  +     * @return The set of Events to be forwarded to the next extension, or core engine if this is the final extension.
  +     */
  +    TreeSet<Event> processEvents(TreeSet<Event> events);
  +
  +}
  +----
  +
  +== Events Aggregation Extension
  +
  +The Events Aggregation Extension allows to scope Sliding Windows on _Events_ and define expressions on aggregated data.
  +
  +To use this feature a _Trigger_ must have the _HawkularExtension_ tag with value _EventsAggregation_. It must then
  +define an _ExternalCondition_ with the _alerterId_ set to _EventsAggregation_, as shown in the example:
  +
  +[source,json]
  +----
  +{
  +  "triggers":[
  +    {
  +      "trigger":{
  +        "id": "marketing-scenario",
  +        "name": "Marketing Scenario",
  +        "description": "Detect when a customer buys several items in a short period of time",
  +        "severity": "HIGH",
  +        "enabled": true,
  +        "actions":[
  +          {
  +            "actionPlugin": "email",
  +            "actionId": "notify-to-marketing"
  +          }
  +        ],
  +        "tags":{
  +            "HawkularExtension":"EventsAggregation"
  +        }
  +      },
  +      "conditions":[
  +        {
  +          "triggerMode": "FIRING",
  +          "type": "EXTERNAL",
  +          "alerterId":"EventsAggregation",
  +          "dataId": "marketing",
  +          "expression": "event:groupBy(context.accountId):window(time,10s):having(count > 2)"
  +        }
  +      ]
  +    }
  +  ],
  +  "actions":[
  +    {
  +      "actionPlugin": "email",
  +      "actionId": "notify-to-marketing",
  +      "properties": {
  +        "to": "marketing@hawkular.org"
  +      }
  +    }
  +  ]
  +}
  +----
  +
  +All events tagged with _HawkularExtension_=_EventsAggregation_ will be filtered out and processed asynchronously by
  +the extension applying aggregated rules defined in the ExternalCondition expressions.
  +
  +=== Events Aggregation Expressions
  +
  +An _ExternalCondition_ used for _EventsAggregation_ alerter defines a DSL expression which is parsed internally by the
  +extension into a DRL format understandable by the JBoss Rules CEP engine.
  +
  +The DSL expression defines Event grouping by fields and additional filtering options:
  +
  +[source]
  +----
  +<expression> ::= "event:groupBy(" <field> ")" [ ":window(" <window> ")" ] [ ":filter(" <filter> ] [ ":having(" <having> ")" ]
  +
  +<field> ::= [ "tag." | "context." ] <field name>
  +
  +<window> ::= ( "time," <time_value> | "length," <numeric_value> )
  +
  +<time_value> ::= [ <numeric_value> "d" ][ <numeric_value> "h" ][ <numeric_value> "m" ][ <numeric_value> "s" ] [ <numeric_value> [ "ms" ]]
  +
  +<filter> ::= <drools_expression>
  +
  +<having> ::= <drools_expression>
  +----
  +
  +For example, the expression
  +
  +[source]
  +event:groupBy(context.accountId):window(time,10s):having(count > 2)
  +
  +can be described as follows
  +
  +[source]
  +groupBy(context.accountId)      Group window events by context "accountId" field
  +window(time,10s)                Define a sliding time window of 10 seconds
  +having(count > 2)               Define an expression on the grouped events
  +
  +In other words, this condition will be true, each time that there are more than two events with the same _accountId_ in a 10 seconds window.
  +
  +The DSL can operate on Events fields, as well as context and tags, as it is shown in the previous example and here:
  +
  +[source]
  +event:groupBy(tags.accountId):window(time,10s):having(count > 1, count.tags.location > 1)
  +
  +where
  +
  +[source]
  +groupBy(context.accountId)                    Group window events by context "accountId" field
  +window(time,10s)                              Define a sliding time window of 10 seconds
  +having(count > 1, count.tags.location > 1)    Define an expression on the grouped events
  +
  +This condition will be true when there are more than 1 events with more than one _location_ tag, so detecting when
  +events for the same _accountId_ happens from different places.
  +
  +The two previous expressions group all events for the timing window.
  +
  +We might have scenarios where only specific events should be grouped.
  +
  +For these cases we can add filters into the expressions like in the following example:
  +
  +[source]
  +event:groupBy(tags.traceId):filter((category == "Credit Check" && text == "Exceptionally Good") || (category == "Stock Check" && text == "Out of Stock")):having(count > 1, count.tags.accountId == 1)
  +
  +This expression will group events filtered by an expression
  +
  +[source]
  +filter(
  +    (category == "Credit Check" && text == "Exceptionally Good") ||
  +    (category == "Stock Check" && text == "Out of Stock")
  +)
  +
  +Note that this expression doesn't define an explicit sliding time window, so it will use a default expiration window.
  +
  +Additional details can be consulted on the JavaDoc of the implementation and examples:
  +
  +* link:https://github.com/hawkular/hawkular-alerts/blob/master/hawkular-alerts-engine-extensions/hawkular-alerts-events-aggregation/src/main/java/org/hawkular/alerts/extensions/Expression.java[Events Aggregation Expression JavaDoc]
  +* link:https://github.com/hawkular/hawkular-alerts/tree/master/examples/events-aggregation[Events Aggregation Expression Examples]
  +
  +== Elasticsearch Integration
  +
  +=== Elasticsearch Alerter
  +
  +The Elasticsearch Alerter will listen for triggers tagged with "Elasticsearch". The Alerter will schedule a
  +periodic query to an Elasticsearch system with the info provided from the tagged trigger's context. The Alerter
  +will convert Elasticsearch documents into Hawkular Alerting Events and send them into the Alerting engine.
  +
  +The Elasticsearch Alerter uses the following conventions for trigger tags and context:
  +
  +[cols="^2,10"]
  +|=======================
  +| *Required* |
  +_trigger.tags["Elasticsearch"] = "<any description>"_ +
  + +
  +An _Elasticsearch_ tag is required for the alerter to detect this trigger will query to an Elasticsearch system. +
  +Value is not necessary, it can be used as a description, it is reserved for future uses. +
  + +
  +i.e.
  + +
  +trigger.tags["Elasticsearch"] = ""                          // Empty value is valid +
  +trigger.tags["Elasticsearch"] = "OpenShift Logging System"  // It can be used as description
  +| *Required* |
  +_trigger.context["timestamp"] = "<timestamp field>"_ +
  + +
  +This defines the timestamp field name for the fetched Elasticsearch documents. +
  +The timestamp field is used to fetch documents on an interval basis, without overlap. +
  + +
  +If there is not defined a specific pattern under the trigger.context["timestamp.pattern"] it will follow the default patterns: +
  + +
  +"yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ" +
  +"yyyy-MM-dd'T'HH:mm:ssZ"
  +| *Required* |
  +_trigger.context["mapping"] = "<mapping_expression>"_ +
  + +
  +A mapping expression defines how to convert an Elasticsearch document into a Hawkular Event. +
  + +
  +Mapping expression syntax (BNF): +
  + +
  +<mapping_expression> ::= <mapping> \| <mapping> "," <mapping_expression> +
  + +
  +<mapping> ::= <elasticsearch_field> [ "\|" "'" <DEFAULT_VALUE> "'" ] ":" <hawkular_event_field>
  + +
  +<elasticsearch_field> ::= "index" \| "id" \| <SOURCE_FIELD> +
  + +
  +<hawkular_event_field> ::= "id" \| "ctime" \| "dataSource" \| "dataId" \| "category" \| "text" \| "context" \| "tags" +
  + +
  +A minimum mapping for the "dataId" is required. +
  +If a mapping is not present in an Elasticsearch document it will return an empty value. +
  +It is possible to define a default value for cases when the Elasticsearch field is not present. +
  +Special Elasticsearch metafields "_index" and "_id" are supported under "index" and "id" labels. +
  + +
  +i.e.
  + +
  +trigger.context["mapping"] = "level\|'INFO':category,@timestamp:ctime,message:text,hostname:dataId,index:tags" +
  +| *Optional* |
  +_trigger.context["interval"] = "[0-9]+[smh]"_ +
  + +
  +Defines the time interval between queries of the Elasticsearch system. +
  +If not provided the default is "2m" (two minutes). +
  + +
  +i.e.
  + +
  +trigger.context["interval"] = "30s" +
  + +
  +performs a new query 30 seconds after completing the previous query, fetching documents generated since that time, using the timestamp field provided in _trigger.context["timestamp"]_.
  +| *Optional* |
  +_trigger.context["timestamp.pattern"] = "<date and time pattern>"_ +
  + +
  +Defines a new time pattern for the trigger.context["timestamp"]. It must follow supported formats of link:https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html[java.text.SimpleDateFormat].
  +If it is not present, it will expect default patterns: +
  + +
  +"yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ" +
  +"yyyy-MM-dd'T'HH:mm:ssZ"
  +| *Optional* |
  +_trigger.context["index"] = "<elastic_search_index>"_ +
  + +
  +Defines the index or indexes where the documents will be queried. If not defined the query will search under all defined indexes. +
  +Elasticsearch wildcards are supported.
  +| *Optional* |
  +_trigger.context["filter"] = "<elastic_search_query_filter>"_ +
  + +
  +By default the Elasticsearch Alerter performs a range query over the timestamp field provided in the alerter tag. +
  +This query accepts an additional filter in Elasticsearch format. The final query should be built from: +
  + +
  +{"query": {"constant_score": {"filter": {"bool": {"must": [<range_query_on_timestamp>, <elastic_search_query_filter>] }}}}}
  +| *Optional* |
  +_trigger.context["url"]_ +
  + +
  +Elasticsearch url can be defined in several ways in the alerter. +
  +See <<Elasticsearch Security>>. +
  + +
  +In addition to the methods defined, a trigger can overwrite the url to which it connects. +
  +Url can be a list of valid link:https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpHost.html[org.apache.http.HttpHost] urls. By default it will point to +
  + +
  +trigger.context["url"] = "http://localhost:9200" +
  + +
  +i.e. a valid url could be
  + +
  +trigger.context["url"] = "http://host1:9200,http://host2:9200,http://host3:9200"
  +
  +|=======================
  +
  +A complete example of an Elasticsearch trigger:
  +
  +[source,json]
  +----
  +    {
  +      "trigger":{
  +        "id": "trigger-project",
  +        "name": "Project Logging Trigger",
  +        "description": "Alert on Project Logging (EFK infrastructure)",
  +        "severity": "HIGH",
  +        "enabled": true,
  +        "tags": {
  +          "Elasticsearch": "Demo ES instance"
  +        },
  +        "context": {
  +          "timestamp": "@timestamp",
  +          "interval": "30s",
  +          "index": "project.logging*",
  +          "mapping": "type|'Unknown':category,@timestamp:ctime,message:text,hostname:dataId,index:tags"
  +        },
  +        "actions":[
  +          {
  +            "actionPlugin": "elasticsearch",
  +            "actionId": "write-full-alert"
  +          },
  +          {
  +            "actionPlugin": "elasticsearch",
  +            "actionId": "write-partial-alert"
  +          },
  +          {
  +            "actionPlugin": "email",
  +            "actionId": "email-to-admins"
  +          }
  +        ]
  +      },
  +      "conditions":[
  +        {
  +          "type": "EVENT",
  +          "dataId": "192.168.122.198",
  +          "expression": "category == 'response'"
  +        }
  +      ]
  +    }
  +----
  +
  +=== Elasticsearch Action Plugin
  +
  +This plugin processes Actions writing Event/Alerts into an Elasticsearch system. +
  + +
  +The Elasticsearch plugin supports the following properties: +
  + +
  +[cols="^2,8,^2", options="header"]
  +|=======================
  +| Property |
  +Description |
  +Default value
  +| *url* |
  +See <<Elasticsearch Security>>. +
  + +
  +Indicate the Elasticsearch server or servers to connect. +
  +|
  +_http://localhost:9200_
  +| *index* |
  +Indicate the index where the Events/Alerts will be written.
  +| _alerts_
  +| *type* |
  +Define the type under the index where the Events/Alerts will be written.
  +| _hawkular_
  +| *transform* |
  +Define an optional transformation expression based on JOLT Shiftr format to convert an Event/Alert into a custom JSON format. +
  + +
  +i.e. +
  +{"tenantId":"tenant","ctime":"timestamp","dataId":"dataId","context":"context"} +
  + +
  +link:https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/main/java/com/bazaarvoice/jolt/Shiftr.java[JOLT Shiftr JavaDoc] +
  +link:http://jolt-demo.appspot.com/[JOLT Online tool]
  +| -
  +| *user* |
  +See <<Elasticsearch Security>>. +
  + +
  +Username for Basic credential authentication.
  +| -
  +| *pass* |
  +See <<Elasticsearch Security>>. +
  + +
  +Password for Basic credential authentication.
  +| -
  +| *forwarded-for*
  +|
  +See <<Elasticsearch Security>>. +
  + +
  +Used for X-Forwarded-For HTTP header.
  +| -
  +| *proxy-remote-user*
  +|
  +See <<Elasticsearch Security>>. +
  + +
  +Used for X-Proxy-Remote-User HTTP header
  +| -
  +| *token*
  +|
  +See <<Elasticsearch Security>>. +
  + +
  +Used for Bearer HTTP authentication
  +| -
  +| *timestamp.pattern*
  +|
  +Alerts and Events use timestamps format for _ctime,stime,evalTimestamp,dataTimestamp_ fields. +
  +If present, this property defines the output pattern of these fields when using a declared *transform*. +
  +It must follow supported formats of link:https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html[java.text.SimpleDateFormat].
  +| -
  +|=======================
  +
  +Examples of Elasticsearch actions:
  +
  +[source,json]
  +----
  +  "actions":[
  +    {
  +      "actionPlugin": "elasticsearch",
  +      "actionId": "write-full-alert",
  +      "properties": {
  +        "index": "alerts_full"
  +      }
  +    },
  +    {
  +      "actionPlugin": "elasticsearch",
  +      "actionId": "write-partial-alert",
  +      "properties": {
  +        "index": "alerts_summary",
  +        "timestamp.pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ",
  +        "transform": "{\"tenantId\":\"tenant\",\"ctime\":\"timestamp\",\"text\":\"trigger\",\"context\":{\"interval\":\"fetch-interval\"},\"evalSets\":\"details\"}"
  +      }
  +    }
  +  ]
  +----
  +
  +=== Elasticsearch Security
  +
  +Hawkular Elasticsearch integration is supported on link:https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html[Elasticsearch REST client]. +
  + +
  +There are several ways to configure a secure connection between Hawkular Alerting and Elasticsearch: Basic authentication, certificates or tokens. +
  + +
  +Basic authentication can be defined on triggers and actions as described in the previous sections.
  +For certificates or token authentication is recommended to use the following system properties or ENV variables. +
  + +
  +
  +[cols="^2,^2,6", options="header"]
  +|=======================
  +| System property | Environment variable | Description
  +| javax.net.ssl.trustStore
  +| -
  +| Location of the Java keystore file containing the collection of CA certificates trusted by this application process (trust store). +
  + +
  +See link:http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html[JSSE Reference Guide]
  +| javax.net.ssl.trustStorePassword
  +| -
  +| Password to access the private key from the keystore file specified by javax.net.ssl.trustStore. +
  + +
  +See link:http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html[JSSE Reference Guide]
  +| javax.net.ssl.keyStore
  +| -
  +| Location of the Java keystore file containing an application process's own certificate and private key. +
  + +
  +See link:http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html[JSSE Reference Guide]
  +| javax.net.ssl.keyStorePassword
  +| -
  +| Password to access the private key from the keystore file specified by javax.net.ssl.keyStore. +
  + +
  +See link:http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html[JSSE Reference Guide]
  +| hawkular-alerts.elasticsearch-url
  +| ELASTICSEARCH_URL
  +|
  +Indicate the Elasticsearch server or servers to connect. +
  +Url can be a list of valid link:https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpHost.html[org.apache.http.HttpHost] urls.
  +| hawkular-alerts.elasticsearch-forwarded-for
  +| ELASTICSEARCH_FORWARDED_FOR
  +| Used for X-Forwarded-For HTTP header.
  +| hawkular-alerts.elasticsearch-token
  +| ELASTICSEARCH_TOKEN
  +| Used for Bearer HTTP authentication.
  +| hawkular-alerts.elasticsearch-proxy-remote-user
  +| ELASTICSEARCH_PROXY_REMOTE_USER
  +| Used for X-Proxy-Remote-User HTTP header.
  +|=======================
  +
  +== REST API
  +
  +Hawkular Alerting supports a robust REST API for managing Triggers, Alerts and Events.
  +
  +* link:http://www.hawkular.org/docs/rest/rest-alerts.html[Hawkular Alerting REST API]
  +
  
positive.code: * Aerogear integration
  * File-system notification
  * Webhook notification
  
  
  === Trigger Dampening
  
negative.code: 
  Note that the string data sent in has any value the external alerter system wants it to be.  In our example it may  be a sensorId and temperature, like "Sensor 5368, temperature 212F".
  
  == Action Plugins
  
  Plugins are responsible to execute actions when an alert, or possibly an event, happens.
  
  