   Best file format for really huge 2D tile maps?
  So there is a game I'm writing code for and it's supposed to have only one really big 2D (as in maybe up to 50000 x 50000) Map and since it would require several hundred megabytes of memory I split the whole thing into several smaller (like 50x50) maps and load only the nearest sub-maps. Memory-usage-wise it's working quite well but I'm wondering what would be the best way to load them from a file? Would it be better to have one large file or several smaller files?   
  That's a huge grid...Would it not be better to use procedural generation then? (Sort of how Dwarf Fortress has HUGE worlds)Some maths i did, assuming you went for minimalistic data:1 Type2 Height3 Bytes((50000*50000)*3)/1024/1024=And that is a whopping 7153 Mb. (7.4 Gb)   ;  Depending how the project goes we could use procedural generation but I'm asking about the best way to store the map data in case we decide against procedural generation   ;Quote:Original post by superschoolshooterWould it be better to have one large file or several smaller files?It doesn't matter.It might be easier to send an update, of one of those smaller regions, with small files.It's also probably a lot better if your regions doesn't have a predefined max-size, since you would have to move all the later regions forward if you increase the size of one region in the middle (provided you want to keep them in order).   ;  What you're doing is probably pretty close to optimal in terms of your in-memory representation. In terms of the file itself, I would organize it in chunks like you have, and apply some simple compression scheme -- RLE or maybe a lossless form of vector quantization -- since the actual disk access itself is going to be your bottleneck a simple compression scheme will actually decrease the amount of time it will take to load a chunk of the map.In terms of what you've got in memory, I'd have 5x5 screens worth of data, which gives you plenty of time to load adjacent pages when you move toward the edges.You could also employ a graph system where each screen of data is a node and the ability to move from one node to another is represented as an edge. If you encode this into your map format it can be used to keep map sizes down by simply not including screens which are entirely inaccessible (can never be viewed) -- for example if you had a very wide impassible mountain range or large bodies of water you have no means to boat around in -- or you might elect to have a special node encoding which means "this screen is all tile N."   ;  What kind of game is this that you need such a big map? And who's going to draw it all?If it's mostly empty space for a space game, you should look into some kind of sparse array representation. Or maybe just remember locations of interesting things and assume everything else to be empty.   ;  well, how are you going to *load* all this data? how you answer that will give you an idea on what youll have to do regarding your data structure :)   ;  50,000x50,000 is to large1 byte per tile = 2.5GB (uncompressed)plus youve gotta add info on whats exactly on the tiles(edit) already posted by ChJees   ;  If you are considering raw uncompressed map data, then yes anything over 1GB (or honestly 500MB) is too large for a 2D tile engine.  I've been pondering this issue for a while with my programmer buddy; we're considering some simple compression like ZRLE for the map files which are also split into chunks (like the original poster described) to allow easy paging into memory.superschoolshooter -- What method are you using to display the visible map regions?  I'm considering using quadtrees on each map chunk, then a simple bounding box for the visible area, and rendering the quads that fall into that visible box.. Not sure how effective this approach will be sadly.  The primary issue I always seem to run into is how to handle tile rendering when you cross map bounds, hopefully the quadtree approach will solve that issue.-Nate   ; While reading this thread I was going over in my head getting each tile down to one byte by splitting up the bits in one single byte to represent different aspects of the tile.. Then I read zedz post saying that 1 byte tiles is still huge. So... I remembered an idea I had for compressing video for streaming, which basically was simply applying a ROAM-esque algorithm to the pixel data.. I'd do something like dividing up the world recursively, dividing each quadrant unless there is no extra detail to store. A quad-tree, basically. What would prevent a quad from subdividing is the fact that it just contains a vast unchanging area, like water, or grass, or desert/etc.. But, then, who is going to design all this data?...   