   Loading array with dynamic sub-array's from a file
  I was wondering how to properly structurize a set of nodes. In my game, sectors are loaded in the background while the players moves around. From what I know, it's best to keep the chunks as large as possible. While the HD is reading, the other threads can continue with the game. If that is true, it would be best to allocate the whole thing in one step, then read the data into the record-array.This time I need to find a way to store a large set of (pathfinding) nodes. Something like<Delphi code>TRelation = record  targetNode : pointer; // pointer to another TNode  data  : integer; // Some additional data stuff  ...end;TNode  = record  relations: array of TRelation; // dynamic array  relationCount : integer;  // count of related nodesend;nodes : array of TNode; // The array I'd like to write/read from a fileSo, there is an array of nodes with a known side. But the relation count could differ for each node. Also loading and storing pointers doesn't work I guess, so I'll have to replace the "targetNode : pointer" with "targetNodeIndex : integer", which is an index in the "nodes" array.I could read node by node, and allocate the "relations" array for each node   stream.read( nodeCount, 4 );   // read the count of nodes   setLength( nodes, nodeCount ); // Allocate node array   // Now start reading node by node   for i:=0 to nodeCount-1 do begin stream.Read( nodes.relationCount, 4 );  // Read relation count into node setLength( nodes.relations, nodes.relationCount ); // Read relations stream.read( nodes.relations, sizeOf(TRelation) * nodes.relationCount );   end;Works, but something tells me it can be done better. And when continously loading/unloading sectors, won't this fragmentate the memory?Greetings,Rick   
Quote:Original post by spekI was wondering how to properly structurize a set of nodes. In my game, sectors are loaded in the background while the players moves around. From what I know, it's best to keep the chunks as large as possible. While the HD is reading, the other threads can continue with the game. If that is true, it would be best to allocate the whole thing in one step, then read the data into the record-array.Windows provides asynchronous IO via Overlapped IO structures. That would be the "best" way to do it.Quote:This time I need to find a way to store a large set of (pathfinding) nodes.More than 20 megabytes? If not, then just load it once, it's not worth the complications.Quote:Works, but something tells me it can be done better.Not really. It's just serialization.Quote:And when continously loading/unloading sectors, won't this fragmentate the memory?No.   ;  20 MB...I whish:) All the nodes together might eat 1 gig. There are millions of them. However, I only load the sectors nearby the player, so probably I only need a few MB at a time. But due lots of loading/unloading, I was afraid of not doing it right.The loading happens in the background in another thread. The loading doesn't have to be superfast. More important is not to block the game threads too much. From what I understand, other threads can just continue while reading. But in the code I gave, the reading is continuously interrupted with reading small bits and setting array lenghts. Maybe its not so bad, but if there is a better way...But if I get your reply right, there's not really a better way to load? I'm not familiar with "Overlapping IO structures". I did a quick read on MSDN, but I don't think its really helping in this particular situation. But I could be wrong of course.   ;  Look into hierarchical pathfinding. Doing any pathing on "millions" of nodes is going to be insanely slow, so you get two bonuses: you don't have to load the lower, finer-grained portions of the graph unless you absolutely need them for pathing; and pathing calculations are automatically faster because you reduce the size of the working set of nodes.   ;  Well, there actually are two levels above this one for "global pathfinding". The millions of nodes will never be loaded or used at the same time. But when summing everything together, this is the estamated node count. One of the ideas was just to load all nodes initially, in order to save headaches with background loading, allocating memory, etc. But in this case the world is too big to load everything at once.   