I am currently (re)designing an event correlation engine and found myself to be uncertainabout the design for the events themselves. I have come up with two possible solutions, but both of them seemsomewhat suboptimal/flawed to me. I have thought about this for some time now, buti have yet to come up with anything better.Problem descriptionThe application takes measurement results from multiple sensors in the real world as input, parses them andcreates events. There are several event classes, each with its unique set of domain specific attributes and methods.Every event, regardless of its class, has a unique id, a timestamp and it can have at most oneparent as well as an arbitrary number of child events. The application correlates the events createdby parsing the sensor data, thus creating a hierarchical event-tree. When a new child is added to an event it may in some cases be necessary to change some attribute values of the parent.Here is a schematic class diagram of the events:                                --------------------------                                |          Event         |                                --------------------------                                | id: int                |                                | parent: 0..1           |                                | children: 0..n         |                                | timestamp: date        |                                --------------------------                                            ^                                            |                  --------------------------------------------------------                  |                                                      |     --------------------------                             --------------------------     |         A-Event        |                             |        B-Event         |     --------------------------                             --------------------------     | < domain specific >    |                             | < domain specific >    |     --------------------------                             --------------------------All events currently being processed by the application are stored in the EventStore.EventStore maintains several indices and methods to effectively search forevents with specific attributes, specific class, ...                                --------------------------                                |       EventStore       |                                --------------------------                                | events: Hashtable(id)  |                                | events_index           |                                | events_tree: B-Tree(id)|                                --------------------------                                | iterate_events(...)    |                                --------------------------The question is, whether it is better to store child events directly in their respective parent events or to store them in the EventStore and reference them by id.Usage example for clarificationOne possible usage for this system is building a SIEM (https://en.wikipedia.org/wiki/Security_information_and_event_management): Say we got a network with multiple hosts, each running a number of services, accessible from the internet (a typical company DMZ). Between the internet and the DMZ there is a firewall which is monitoring network traffic to the internet, while the switch monitors network traffic within the network. All messages (infos, warnings, alerts, ...) created by both the firewall and the switch are sent to the SIEM. The SIEM parses this data and creates events. Some possible examples for events are:PortScanEvent: Created when the firewall recognizes a PortScan. Possible attributes are scan_type (TCP SYN scan, TCP Connect scan, ...), source (where the scan apparently originated from), ...ARPCachePoisoningEvent: Created when the switch recognizes a ARP cache poisoning attack. Possible attributes are redirected_from (whose traffic got redirected), redirected_to (where to the traffic got redirected), ...SSHAccessEvent: Created if somebody tries to connect over SSH to one of the hosts. Possible attributes are: target_host, source, target_ports, successful, .... SSHAccessEvent might also have a method called attack that determines whether the connection attempt is likely to be a attack or just a regular connection by one of the administrators (> Some events have methods to further process the data associated with the event)After parsing the sensor data and creating events the correlator module does its work: If the event correlation module for example sees a PortScanEvent to port 22, shortly followed by a SSHAccessEvent it might decide to correlate both events into a MaliciousSSHAccessEvent (the PortScanEvent and the SSHAccessEvent become children of the new MaliciousSSHAccessEvent). This can be done not only once, but an arbitary number of times, meaning the MaliciousSSHAccessEvent itself could be correlated together with another event into an even higher event.Possible solutions1. Store child events directly in the parent event.Child events are stored directly in the parent event. When a child is added to an event, the parent events changes its own attribute values as required and inserts the child.The event store does not know about child events, it only knows the current top level events.                                --------------------------                                |          Event         |                                --------------------------                                | id: int                |                                | children: Event: 0..n  |                                | timestamp: date        |                                --------------------------                                | add_child(event)       |                                --------------------------                                            ^                                            |                  --------------------------------------------------------                  |                                                      |     --------------------------                             --------------------------     |         A-Event        |                             |        B-Event         |     --------------------------                             --------------------------     | < domain specific >    |                             | < domain specific >    |     --------------------------                             --------------------------Pros:It feels natural that a event should know how to add a child toitselfThe add_child method can be used to set the attribute values of the parent as requiredCons:EventStore does not know about child events and therefore cannot include them in the search index it maintains. Since the vast majority of events will be child events this is a real performance hit.One way to work around that would be to give every event a reference to the event store, i do however think that this is a bad idea:It would cause a circular dependency between Event and EventStoreA single event should not have access to, nor know about the event storeProblem:Suppose we have the following event tree and want to add a additional child to event_3.event_1  event_2    event_3      event_4We call event_3.add_child(...) and event_3 will change its attribute values as required and add the new event as its child. The problem is that since the attribute values of event_3 have changed is may be necessary to change the attribute values of event_2 and event_1 as well, but i cannot think of an elegant way of doing that (since we don't have a reference from event_3 to event_2 or event_1). 2. Store all events in the EventStore and reference parent/children by idAll events are stored in the EventStore. Instead of the actual events each event only stores the ids of its parent and its children. The add_child method has to be moved to EventStore: Before a child is added to an event it has to be added to the EventStore, but the event itself cannot do that since it dosn't have a reference to the EventStore.                                --------------------------                                |          Event         |                                --------------------------                                | id: int                |                                | parent: int: 0..1      |                                | children: int: 0..n    |                                | timestamp: date        |                                --------------------------                                            ^                                            |                  --------------------------------------------------------                  |                                                      |     --------------------------                             --------------------------     |         A-Event        |                             |        B-Event         |     --------------------------                             --------------------------     | < domain specific >    |                             | < domain specific >    |     --------------------------                             --------------------------                                ----------------------------                                |       EventStore         |                                ----------------------------                                | events: Hashtable(id)    |                                | events_index             |                                | events_tree: B-Tree(id)  |                                ----------------------------                                | iterate_events(...)      |                                | add_child(parent, child) |                                ----------------------------Pros:EventStore now has direct access to all events, making the search for specific events    more efficientCons:Placing the add_child method in EventStore seems awkward.QuestionI would like to get the best of both  worlds: Have the add_child method in Event, while at the same time referencing children/parent via their ids. Is there some way to accomplish this? I am looking forward to reading your thoughts on the matter.
Event correlation engine design: Nesting events vs. using event ids
design;architecture;language agnostic;coupling;correlation