Python RPC Design Pattern - is bidrectional coupling a good idea? I want to build a barebones system according to the following spec:
Given a setup of n computers, if the ith computer receives an operation to update a given key-value pair, it should perform the update, and propagate it to all other nodes. I'd like communication between nodes to be done with RPC.
I have the following runnable code that works fine:
import aiomas


class RpcCall:
    def __init__(self, port, method):
        self.port = port
        self.method = method

    async def __call__(self, *args, **kwargs):
        rpc_con = await aiomas.rpc.open_connection((&quot;localhost&quot;, self.port))
        rep = await getattr(rpc_con.remote, &quot;redirect&quot;)(*args, **kwargs, method_name=self.method)
        await rpc_con.close()
        return rep


class Node:
    def __init__(self, port, neb_ports):
        self.server = None
        self.port = port
        self.table = {}
        self.neb_ports = neb_ports

    def start(self):
        self.server = aiomas.run(aiomas.rpc.start_server((&quot;localhost&quot;, self.port), Server(self)))

    def close(self):
        self.server.close()
        aiomas.run(self.server.wait_closed())

    async def update(self, key, value, replicate=True):
        self.table[key] = value
        if replicate:
            for neb in self.neb_ports:
                await self.replicate(neb, &quot;update&quot;, key=key, value=value, replicate=False)

    async def replicate(self, node, method_name, *args, **kwargs):
        rpc_call = RpcCall(node, method_name)
        await rpc_call(*args, **kwargs)


class Server:
    router = aiomas.rpc.Service()

    def __init__(self, node: Node):
        self.node = node

    @aiomas.expose
    async def redirect(self, method_name, *args, **kwargs):
        func = getattr(self.node, method_name)
        await func(*args, **kwargs)


if __name__ == &quot;__main__&quot;:
    n1 = Node(5555, [5556])
    n2 = Node(5556, [5555])
    n1.start()
    n2.start()

    aiomas.run(n1.update(&quot;foo&quot;, &quot;bar&quot;))

    print(n1.table)
    print(n2.table)


As the output should show, the table is replicated onto both nodes, ensuring there's no data loss in case one node dies.
I'm looking for a review specifically on the design pattern used in the above.
I've coupled the Node and Server classes so that whenever the Server receives an RPC call, it delegates it back to the Node, which then performs the operation on the underlying table 
Good idea/bad idea?
On the coding side, every Node has an attribute Server and the Server has a reference to the Node it belongs to. Node calls Server's methods, and Server reads Node's attributes, so they currently are inseparable classes. I would suggest refactoring this, and - personally - I would just merge the two classes.
I'm also not sure why RpcCall needs to be a callable class, other than for the sake of enforcing OOP. It looks like a simple function (or another method in Node) would do just as well. Then again, maybe your real scenario needs it to be a separate class.
for neb in self.neb_ports:
    await self.replicate(...)

This line defeats the purpose of asyncronous IO, because you are waiting for each call to finish before starting the next one. You could look into Asyncio.gather to start and await multiple such calls in parallel instead of sequentially.

Regarding reliability, you - unfortunately - have none. If one node fails, it will bring peer-nodes down at the next attempt to sync the key-value store. The reason is that you are awaiting a function that will timeout and raise and Exception if a connection can't be established. You are not handling this exception anywhere, so the calling node will halt. If keys are updated often enough, your entire cluster will die due to this.
If a connection can be established, I'm not sure what will happen if either node dies. I briefly searched aiomas, but couldn't find related documentation. My guess is that the call will just hang indefinitely. This is potentially worse than a clean crash because your node will be leaky and silently suffer a death by a thousand cuts. Log messagees (if any) will likely point you into the wrong direction, making this problem hard to debug.
On the point of fault-tolerant because &quot;there is no data loss because nodes replicate preemptively&quot; it really isn't just that simple.
The sore point is the hierarchy among nodes, or rather the lack thereof. If the network for one node dies temporarily, it won't receive any further updates. To the node, the faulty network and not getting state changes look alike, so it will continue running happily. To the other nodes of the cluster it looks like the node died, and - assuming you add exception handling - they will continue to change state jointly. After the network issue is resolved, the lost node will reappear in the cluster and continue serving happily. However, it is now out of sync and will serve potentially outdated key-value pairs or report missing values for new ones that were set during the network outage.
Assume this happens to multiple nodes; you'll end up with a heterogenious cluster and - because you lack hierarchy among your nodes - it is impossible to say what the correct state should be. Currently, you can't differentiate the order in which state-changes occurred, so rebuilding the state is ambiguous. You will need to define merging and conflict resolution mechanisms to re-establish synced state.
You may also wish to establish a heartbeat for the cluster so that a node can discover connection issues. How it should handle them is context-dependent, but an easy first idea is for the node to simply suicide.
Instead of creating a homebrew design-pattern for this, it may be easier to follow an established consensus method such as RAFT or PAXOS. This will take care of a lot of edge cases and may make your life considerably easier.