   .pk3/.zip archives for game data?
  Is there any IO latency gain in using a single big .pk3/.zip archive to store game data in, rather than using loads of small files? Or why do many Id Software games use this kind of packing of data?   
  Yes since,1: Compression reduces the amount of bytes you need to copy from the hard drive to memory.2: Compiling many small files into larger archive reduces the total seek time.   ;  Some reasons they place all their game assets into an archive are security, compression, and ease of moving. With everything in a single file, even though it is just a .zip file, people cannot access the game files and make easy changes without decompressing it. So the average user would not accidentally delete or move things around and lose some files. With as much game data as a typical id game has, using the compression from .zip helps save on space. This also helps with game installation by having only one large file to transfer instead of tons of smaller files.Now I am not 100% sure on this but I did read it elsewhere and it makes sense. Using one archive requires only one file handle so it reduces system resources. Instead of having several open file handles for sounds, textures, map information, entity information, etc.. you just have one file resource to keep track of. During development I believe they just keep all the files uncompressed since items are constantly being added and changed.    As far as overhead for using it I am sure some exists but in a typical id game the level load time is when the look up and decompression occurs for most of the information so a small delay between levels is not a bid deal. I am not sure if they actually do any during game play.   ;  There are several reasons for the .pk3 .zip .wad etc file types.1) Compression saves you disk bandwidth at the cost of compute resources. Compressed files are less bytes to read off the disk, and disk reads and seeks are SLOW compared to a fast compression scheme (LZ varient like in .zip). So, you spend all your time waiting on the disk. Less bytes on disk means less time waiting for a load.2) as mentioned above, it can also reduce seek time to have a lot of small files packed into one big file.3) You can go a step further, and if you have a pipeline to build these .zip files, they can contain transformed data.  Instead of doing transforms on your data at load time, you do them all at packing time.  For instance, you'd convert .psd, and .wav files to .tga and .mp3 files at packing time. You'd take your lua scripts, and run them through a lua compiler and store the binary lua script instead of the text one.  Thus your source assets are in the easy-to-work form, and you never have to remember to save them in the game-friendly form, and your game never has to load anything besides the optimal formats.4) You can have a VFS system in your game, so it CAN read packed or unpacked data so that you don't have to pack up data into a .zip just to run the game. The same VFS also makes it trivial to implement a "patch" or "mod" system, where in any files in "mod.zip" that have the same path as "game.zip" will override that file.See the PhysFS code base.5) The grand step that many games make is storing ready-to-use data in the .pak files. So, after decompressing the data, they can just cast it to a structure (and fix up pointers).  Thus the deserialization process doesn't involve any data transformations. The easiest way to look at this is the difference between loading a model file that is stored as a properly packed set of data you can just pass as an OpenGL vertex array pointer. vs having a text file that you parse to build said vertex array.;Quote:Original post by KazeYes since,1: Compression reduces the amount of bytes you need to copy from the hard drive to memory.If the CPU can compress the data faster than it can be loaded from the hard drive/DVD. It depends on the platform and the compression algorithm.Quote:Original post by Kaze2: Compiling many small files into larger archive reduces the total seek time.Putting them in the order they are going to be accessed reduces seek time even more. Especially on DVDs.   ;Quote:Original post by RattenhirnQuote:Original post by Kaze2: Compiling many small files into larger archive reduces the total seek time.Putting them in the order they are going to be accessed reduces seek time even more. Especially on DVDs.Unless your making a console game trying to control the physical mapping of individual files doesn't seem like it would be worth the effort.   ;  I am curious about something since I started looking at the pk3 game files shipped with Quake 3... Why does id keep many pk3 files then? and why do all of them contain a copy of the virtual machine files? does anyone have any idea?;  One reason for multiple .pk3s was for the free demo. They can put just the data needed for the demo in one .pk3 to distribute and then when you purchased the full game the rest of the data was loading in other .pk3 files. ;Quote:Original post by LothlorOne reason for multiple .pk3s was for the free demo. They can put just the data needed for the demo in one .pk3 to distribute and then when you purchased the full game the rest of the data was loading in other .pk3 files.Yes, but there's 9 of them, pak0.pk3 being the biggest at 480MB, the others range between 270kb and 8MB [tears]   ;  I wonder if it was due to a file size limitation on one of the systems Quake3 supported then since they seem to be keeping them below 500mb.   