 node, then the message may be sent on all of the node's links. Cycles may be eliminated similarly as for the broadcast on all links operation. One exemplary use of the broadcast on role routes operation is to recover routes to the role.
When a node joins a network, the network may already have trees unless the network is new. In one embodiment, the following process may be performed for a node to get on all trees. First, the node may broadcast a “Got trees?” message using the broadcast on all links operation described above. If no response is received within a given timeout interval, then the process may be done (since there are no trees). FIGS. 42-49 illustrate a tree building process when a group of nodes joins a network, and a tree spanning the nodes is built.
If it is determined that there are trees and the node is not on all trees, then the node may request all trees from each neighbor. If not on all trees, each neighbor may in turn request all trees from its neighbors in a recursive manner. Cycles may occur, but only one request to each neighbor is performed. Once a node is on all trees, the node may supply all the trees to each requesting neighbor. A receiver of trees may receive some trees from each neighbor to avoid getting all trees over one link. FIGS. 50-57 and 58-71 illustrate this process.
In one embodiment, hop timing may be used to compute timeouts during the tree building process. A local hop time may be computed as a running weighted average of local ping times. In one embodiment, each ping affects 10% of the next computed local hop time and the previous local hop time affects 90% of the next computed local hop time. The ping rate may be configurable. In one embodiment, pings may be performed once per minute. The local hop time may be piggybacked on every builder message.
A global hop time may be computed based on the local hop times. In one embodiment, the piggybacked local hop time affects 10% of the next computed global hop time, and the previous global hop time affects 90% of the next computed global hop time.
Timeouts for the tree building process may be computed as a function of the maximum number of expected total remaining hops and the global hop time.
As described above, in one embodiment of the T&R layer, a message routing engine 134 may manage the routing of messages. The message routing engine on a particular node may be invoked by a client application using an application programming interface (API) or may be invoked in response to receiving a message via a link.
Client applications which receive a message from a sender may reply to the message with one or more responses. The response(s) may be routed back to the sender over the same route that was used to send the message. The API for sending a response may include parameters specifying the ID of the message being responded to, the response (e.g., an array of bytes and size of the array), as well as parameters specifying various options.
In one embodiment, the concept of a session may be utilized to allow a message sender to receive multiple responses to a message. A “last reply” Boolean value (e.g., a value included in the response header or a parameter passed when sending a response) may be set to True when the last response to the message is sent. FIG. 72 illustrates an exemplary session. As shown, a sender sends a message to a receiver. The receiver sends four response messages back to the sender. In the fourth response message, “last reply” is indicated.
In one embodiment, the T&R layer may support aggregate responses such that the sender receives a single response message which includes all responses from all receivers. The client application listener on the sender may not be invoked until all responses have been received. Thus, the client application may synchronize with multiple responses.
In another embodiment, the sender may receive each response as a separate response message. In one embodiment, the T&R layer may support one-way messaging so that responses by receivers of a message are not allowed. In one embodiment, each message may be handled using the messaging technique desired by the sender, e.g., aggregated responses, separate responses, or no responses.
In one embodiment, responses may flow back to the message sender over the original path by which the message was received. FIG. 73 illustrates an exemplary network in which a message is sent from a sender node 320 to a receiver node 321. For example, the message may be addressed to a role on the receiver node 321. The path of the message is illustrated. As shown in FIG. 74, a reply sent by the receiver node 321 is sent over the same path.
FIG. 75 illustrates an example in which a message is sent from a sender node 320 to multiple receiver nodes 321A, 321B, and 321C. For example, each receiver node may have an instance of a particular role to which the message is addressed. As indicated in FIG. 76, each receiver node may reply to the message received. FIGS. 77-83 illustrate a technique according to one embodiment for sending the responses from the receiver nodes back to the sender node 321. As illustrated in this example, the responses are aggregated as described above so that the sender node 321 receives all responses in a single response message. In another embodiment the sender node 321 may receive three separate response messages (or more than three if one or more of the receiver nodes sends multiple responses). However, aggregating the responses may help to conserve network bandwidth.
FIG. 84 illustrates a network in which nodes are grouped into two partitions. Each node which has an instance of a particular role is indicated with a colored circle.
As another example, the response may be sent with a “last reply” parameter indicating that the response is the last reply to the message, as described above. Any given recipient of the original message may set the “last reply” parameter to True only once.
In one embodiment the T&R layer may change the “last reply” parameter in some situations. For example, if “last reply” is set to True, a node forwarding the response along the route may change “last reply” to False if the node has an outstanding link on which it has not yet received a response with “last reply” set to True or if the node has not yet received a response from a local client (a client application on that node which received the original message from the sender) with “last reply” set to True. This ensures that the sender receives only one response with the “last reply” parameter set to True, even though multiple responses may originally be sent from receivers of the message with the “last reply” parameter set to True. In another embodiment, the sender may always receive response messages having the original “last reply” parameter values set by the respective recipients, and the sender may keep track of which recipients it has received a last reply from and which it has not.
As noted above, when a recipient node with a role instance issues a reply with a “give up role” parameter set to True, the role instance may move to the sender node. In one embodiment, this may be accomplished by performing an un-publish operation to remove the role instance from the recipient node, followed by a publish operation to add the role instance to the sender node. However, in another embodiment a more efficient technique of moving the role instance to the sender node may be utilized. The more efficient technique is based on the observation that only routes maintained by nodes along the path of reply (which is the same as the path over which the original message was sent) need to change. Thus, each time after the reply with the “give up role”=True parameter is forwarded by a node, the route on that node may be re-pointed to point in the direction in which the reply was forwarded. Thus, the next time that node receives a message addressed to the role, the message may be routed in the direction of the node which now has the exclusive instance of the role (i.e., in the direction of the original sender node which requested the role).
FIG. 85 illustrates a network including a node 330 with an exclusive instance of a role. Routes to the role instance are illustrated by the arrows. FIG. 86 illustrates the route of a message sent from a node 331 to the node 330 (route indicated by bold arrows), where the node 331 requests to add an instance of the role. The node 330 may send a response message back to the node 331 with the “give up role” parameter set to True. As described above, nodes along the path of reply may change their routes to point in the direction in which the response message is forwarded. FIG. 87 illustrates the route changes (illustrated by the bold arrows) and the new owner of the exclusive role instance.
In various embodiments, any of various kinds events may be generated by the T&R layer in response to certain situations. Client applications may be notified of particular events through application listener ports. The following describes some exemplary events which the T&R layer may utilize.
Tree Built event—indicates that a tree has been constructed or a tree object has been instantiated. The Tree Built event may include information identifying the protocol (e.g., client of the T&R layer) that caused the tree to be created. Thus, applications may learn about new trees by receiving Tree Built events. As described below, in one embodiment an application may create a snooper in response to a Tree Built event.
Role Not Found event—indicates that a message was not delivered to any instance of the role to which it was addressed.
As shown in FIG. 38, in one embodiment the T&R layer may allow client software to act as a snooper. A snooper may intercept messages sent from a sender to a receiver and may intercept any responses sent from the receiver to the sender. The snooper client may be located on any node between