   fastest method to read from files
  Hello everyone,so yeah as the title says, what is the fastest way to read from a file? is it the C wayFILE *fp = fopen ( /*etc*/ );fclose ( fp );or C++ fstream way?what might affect reading speed? like reading it letter by letter, etcwould it be better to read the whole file and close it, and store in a string, and use the data in the string for faster access, or something like this?Thanks for reading,-Hassan   
  IO will always be the bottleneck.For everything else, it depends on the OS, file system, layout of directories and files, type of contents, file system cache, etc....How many gigabytes or tens of thousands of files do you plan on processing?   ;  well im making a file format the includes a whole game's data ( objects / images / etc ), so it will be a really huge file, so i'm looking for the fastest way before i go into details and re-write the whole thing, number of files won't be too much ( 1-3 max ), but they will be huge   ;  Use only one fwrite/fread (or equivalent) call (if possible).   ;  IO is going to be a huge bottleneck. It takes miliseconds to seek and read stuff.Use the minimum number of file calls to minimize IO overhead.Opening a file causes the OS to search the file tables for it. If the table is uncached right now, that means a seek (some consoles don't cache the file table, so it is always a seek).Issuing all your reads at once (one big read, or many async reads) gives the OS a better chance to schedule your reads to minimize seeking.One of the best things you can do on a modern system to increase performance is compress the files. Then thread your application and read in "small" blocks, maybe 1Mb or 2Mb at a time on one thread. In another decompress the blocks as they get read in.  On faster hardware (SSDs or RAID), you might see less gains from this, since you have proportionally less time to decompress each block, but you likely have more processors to offload blocks to for decompression so it can still work out.;  Hi Hassanbasil,when I did some research about this problem (many years ago, so what I say now could be obsolete) I found that "memory mapped" files on Windows were a good way to handle big files. IIRC there was also something on Gamasutra.   ;Quote:Original post by TxkunHi Hassanbasil,when I did some research about this problem (many years ago, so what I say now could be obsolete) I found that "memory mapped" files on Windows were a good way to handle big files. IIRC there was also something on Gamasutra.I read that article as well, it was very helpful.   ;  so, fstream beats old C way? alright gonna try it and see the difference.so i should make one file and add all data to it, and read it at run-time with multiple threads, that sounds interesting, would be great if i get it to work properly.thank you all for your help and ideas.P.S. : can i open a file, for reading and writing at once?   ;Quote:Original post by TxkunHi Hassanbasil,when I did some research about this problem (many years ago, so what I say now could be obsolete) I found that "memory mapped" files on Windows were a good way to handle big files. IIRC there was also something on Gamasutra.That's still current (as of 2-3 years ago :P), memory mapped files are an easy and highly efficient way of dealing with large files, and not only on Windows.   ;Quote:...P.S. : can i open a file, for reading and writing at once?You certainly can. This is how I open my files (I swap in and out data during running my engine, however don't know how well it will work yet (considering load/save times).File = new fstream(  Filename.c_str(), ios_base::in | ios_base::out | ios_base::binary  );