   P2P network concepts
  I know this has not directly to do with games, but I think I might get an answer here.I'm looking into a P2P system for over a local network for transfering large amounts of raw data (read: files of 700+ mb) and I need a system that is very robust. Messages such as searches are transfered through the network, while raw data is transfered over a direct connection.Firstly, a traditional system, where each node knows every other node. Advantages are that it's impossible to break. If one node fails, a node can still contact all other nodes.Disadvantage is that if a search is done the sending node might be stressed a lot. Also very important is that the nr of sockets (or peer data) can grow out of hand if the network grows.So, I need something scalable.Then, I know of a tree-like structure, where each node has a nr of child nodes.Disadvantage is that no node can directly contact another node. So if a direct message has to be sent, it will have to propagate it through the network. It eats more bandwidth, but I have no idea how much nodes I'm looking at at the first place, and how deep the tree can get. With some pathfinding I can limit the times a message is duplicated to DEPTH*2 times, but a dumb node implementation can make it as worse as N times (where N is the nr of nodes).Also, it's not as robust as the previous concept, as if one parent fails, the rest goes down too.Advantage is that it's a lot lighter on resources.Now, my question is, are there any other network concepts?? I've looked at the circular tree structure, but it can't assure that each node can reach all other nodes (ie: two seperate networks existing). So, I'm still looking at my options.Requirements:- each node has to be able to reach every other node (100% assurance)- It has to be robust, I have no hard requirement for reconnection time in case of a node failing, but it has to be a virtually transparant happening for the rest of the network- The number of messages should stay withing acceptable limits. Even though the huge amount of raw data will kind of makes the messages look like peanuts.Thanks!![edited by - Structural on February 18, 2004 2:55:29 AM][edited by - Structural on February 18, 2004 3:09:55 AM]
  Well, if you insist on building it yourself, look into bittorrent. It''s a very robust distributed file transfer network. Instead of having 1 server to spread all the data to the clients, there is a tracker/seeder and clients. The seeder distributes his data over the clients once. So, client 1 gets packet A, client 2 gets packet B and client 3 gets Packet C. When either of the clients is done downloading it''s packet, it will make it avaliavable for the other peers immediatly. It will spread the packet to the other clients, so the seeder has more bandwidth for the other packets. To optimize it more, you could have the seeder spread rare packets aswell when there are several clients with 100% distributed copies. For more information on bittorrent:http://www.bitconjurer.org/BitTorrent/You could implement it yourself, or just use bittorrent internally, and save yourself the hassle.Toolmaker   -Earth is 98% full. Please delete anybody you can. ;  Yes, I insist on implementing it myself, because it''s for an embedded system. Thanks for the tip on bittorrent, but I''m not sure if I can use it.The idea of this new system is to take out the cental server of the network and move this functionality to the clients but maintain a stable network with a 100% assurance of each node being able to reach each other node. So especially the search and network concept are important but I''m also dealing with limited system resources and a good scalability.Do you know any other networks I can look into?? So far I''ve looked at fasttrack, Gnutella, the JXTA platform, and now bittorrent. ;  What are the real-time constraints? It sounds like this is "slurp big files" rather than "keep lots of nodes in sync in real-time".If these are all on the same subnet, you can use broadcast packets. If everyone needs the same data, this is almost ideal. There are existing utilities that do this, such as the enterprise system-image applications that clone out a Windows install to 50 machines (in a classroom, office, etc).There are video distribution methods (for edge-attached video-on-demand, say) which also do something similar; they send out a stream of packets using broadcast; the stream is slightly redundant such that if you get any 80% of the packets (or whatever) you can reconstruct the data.An entirely different topology to think about is regular grids, cubes, hypercubes, etc; probably with wrap-around and the edges for good measure. I don''t know how many nodes you''re talking about, but consider a system of 16 nodes, arranged in a 4x4 grid. Each node talks to its four neighbours. Also, the edge nodes talk to the edge nodes on the other side, making the topology like a torus (or, in the case of more than two dimensions, hyper-donut :-)In this topology, you need to take out two whole rows, or two whole columns, before the network becomes severed in two parts that can''t communicate. If you have 256 nodes, you get a grid 16 to a side, and 32 nodes need to go out before you have severed the grid.Another topology to consider is something akin to a skip list, and make the two or three highest levels talk directy to each other to avoid the problem of the root failing. When inserting a node, you pick a random level (with higher chance for a lower-down level) and insert the node at that level in the "skip list" of systems. Each system talks to its parent, the next/previous at its current level, and its first child. You need at least two systems failing to take out a chain of children.Well, some ideas. There''s actually significant amounts of research out there (after all, this was one of the problems the internet was designed to solve :-) 