   Efficient disk I/O in Windows
  Hi everyone,I have some questions regarding fast disk reading on a Windows server. A lot of people who are working on MMO games or such that require a lot of data retrieval are probably using commercial database management systems. I'm interested however in retrieving large amounts of data using a custom format that would not fit well with a relational DBMS, therefore I want to roll my own high-speed data retrieval code.The information that I'm most interested in is how a modern DBMS, such as SQL Server or Oracle, performs its I/O on the Windows platform nowadays. I studied databases fairly heavily in college and I was always taught that for the most efficient disk reading, you store your data in disk blocks in the order that it will be read (as least as far as you can manage) so that the blocks can be sequentially read with minimal seek / rotational delays, in parallel with the CPU processing of the blocks being read. The Windows file system does not guarantee that anything you write will be in nice blocks (because the actual disk block has NTFS information on top of your written data), or that it will even be in sequential order because the file blocks may be scattered all over the disk, even when they are written in sequence by the program. According to Wikipedia, "Database management systems often use their own block I/O for improved performance and recoverability as compared to layering the DBMS on top of a file system."My question is, do modern Windows DBMS's perform direct block access, or do they go through the file system? Is direct I/O still beneficial in a multitasking operating system where several programs are all using the disk at once? Can it be performed without stepping all over the NTFS file system and corrupting data?These may be tough questions, but I appreciate any guidance that can be provided. Thank you very much!   
  I'd say these days sequential access is much less important for performance, at least on servers where you can control the hardware. That's because as long as your data is on an SSD the performance penalty for non-sequential access is relatively small - for example the best single drives will do 200MB/s of random 4k reads, and more more specialized hardware is even quicker.If you really want to bypass the filesystem, I don't think it'd be a good idea unless you take over whole drives. The defragmenter is likely to mess up any direct access, if nothing else does.   ;Quote:Original post by Adam_42for example the best single drivesApologies for going somewhat off-topic, but having owned one of those (the non-X2, not the very newest generation), I object to the word "best". On a 120GB drive which held 10GB total of infrequently written data, I was seeing occasional "bad block" messages in the event log after 3-4 months. Which, frankly, isn't acceptable, and for which there is really no excuse.I did not actually notice any corrupted data or lost files, but seeing some more or less obscure, frightening failure message without really knowing what broke leaves an even more uneasy feeling.The single basic requirement that a storage device (of whatever kind) must fulfill is that it reliably stores data for a reasonable time under reasonable usage. I would not object if the drive was 5-8 years old or if it was written to 50-60GB per day. Hey, sooner or later, things must break.However, if you planned in a 90% "security marge", then it simply has to be 100% good, and there is no valid excuse nor any way to make up for that after the fact.And with that, back to topic :-)Regarding that Wikipedia quote, this is probably a candidate for "citation needed". While it may be possible that some systems do that, none of the "normal" ones that a million of people use every day does (such as for example one well-known Swedish SQL database, or one well-known Italian no-SQL key-data store, or one well-known Japanese no-SQL key-data store).Writing blocks to disk directly has few advantages if any, but makes management a hell lot more painful, so I would not know why one would want to do that. Files are a good thing.More often than not, sequential raw block acces is indeed of advantage, but as a matter of fact, for the particular case you just don't know. Raw "disk" may be a single contiguous real disk or partition, or a file image mounted as disk, a contigent of 6 disks with different sizes and speeds, or whatever. You might be writing into a snapshot while a backup is running at the same time (this is a very common thing), and you wouldn't know.On the other hand, reading and writing raw blocks is a bitch to maintain compared to files. With files, if you want to make a backup, you take a snapshot, zip your database file and copy it somewhere, discard the snapshot, end of story. If you are using raw disk access, you must zip the whole partition, since you don't know which sectors might be used!And in case your harddisk blows up or you want to upgrade hardware in three years from now, you must create a partition with the same size when restoring, copy the whole partition, and then somehow tell the database program to "enlargen" the data store.Or, you must export and re-import your whole data. Good luck if your data is half a terabyte.With files on a filesystem, you simply unzip what you have. Who gives a crap how big the partition is, it will just work.And having said that, reserving large contiguous areas on the drive so your data is mostly or entirely sequential on-disk is something that every half decent filesystem (including NTFS) supports, and the actual performance overhead when using a high-performance FS (maybe not precisely NTFS) is neglegible.Maybe the Wiki wording is just a bit misleading insofar as some database systems (but not all) will avoid going through the filesystem cache, for various reasons. Maybe that is what they meant.   ;Quote:Original post by JonWThe information that I'm most interested in is how a modern DBMS, such as SQL Server or Oracle, performs its I/O on the Windows platform nowadays.Very carefully and slowly. Each time any piece of data is updated in the database, everything stops until OS confirms that the transaction has been fully committed to disk. So if you update one byte, the database will stop everything and wait for disk to confirm that it has been written correctly. Nothing else matters, correctness and reliability take first place.Quote:so that the blocks can be sequentially read with minimal seek / rotational delaysThat depends on data, not disk. If data and queries performed on it exhibit locality, then disk accesses will be local. If you do update *, then they will be all over the disk.Quote:in parallel with the CPU processing of the blocks being read.Once you hit the disk you're slow already, and parallel can only be done to the degree transactions allow. For web, memcached is used. For large systems (banks and such), big iron is brought in. That one goes beyond 100k transactions per second if the price is right.How to scale RDBMS? For heavy transactional data, buy fastest disks possible, put them in suitable RAID with lots and lots of RAM. For mostly read-only data, memcached. Horizontal partitioning may help as well.Also, do not write your own RDBMS. Existing RDBMS databases have 40 years of top expertise in the field involved in them. Evaluate existing options - there is nothing you can do better.Quote:I'm interested however in retrieving large amounts of data using a custom format that would not fit well with a relational DBMSWhy not? What does the data look like? Format is completely irrelevant - what are access patterns? The rates? The latency? Transactions? Queriability? Relations?Quote:Apologies for going somewhat off-topic, but having owned one of those (the non-X2, not the very newest generation), I object to the word "best". On a 120GB drive which held 10GB total of infrequently written data, I was seeing occasional "bad block" messages in the event log after 3-4 months. Which, frankly, isn't acceptable, and for which there is really no excuse.Which is a moot point. Hardware fails. Whether it fails once a month, once a year or once a decade - it fails. If data is even remotely important, this is addressed in the design via replication, mirroring and backups. Rate of failure is merely a cost factor to consider. But your disk *will* fail, that is as inevitable as death and taxes.   ;Quote:Original post by samothApologies for going somewhat off-topic, but having owned one of those (the non-X2, not the very newest generation), I object to the word "best". On a 120GB drive which held 10GB total of infrequently written data, I was seeing occasional "bad block" messages in the event log after 3-4 months. Which, frankly, isn't acceptable, and for which there is really no excuse.you relying on a single point of failure would be no excuse. but my drives (x1 and x2 and non intels) none behave like you described => your drive might be buggy (or needs a firmware update, there where some for x1 and x2). it might just be a missunderstanding of the drive and the os => nothing at all.but the most important point you said: no data loss ever. just shows those drives work, and work great. i would never move back to platters anytime.and to the op, i would try to get it fit into databases. consider the day that you suddenly need more than one server for scaling. with a database, that's (reasonably) simple. with your own file format, you suddenly have to code all kind of synchronization and all by hand.   ;Quote:Original post by JonWthat would not fit well with a relational DBMSWould one of the non-relational DB's match your requirements, maybe?;  Thanks for the opinions everyone.Quote:Original post by samothMaybe the Wiki wording is just a bit misleading insofar as some database systems (but not all) will avoid going through the filesystem cache, for various reasons. Maybe that is what they meant.I'm considering going the route of asking Windows not to cache my I/Os for me (unbuffered mode), because it is fairly simple to do your own caching (it only adds a few constraints on the sizes of buffers as well as their placement in memory and in the file). This Microsoft Research paper advocates turning off buffering when spreading the I/O operations across several drives -- unbuffered I/O was over 8x faster (there's some cool stuff in Microsoft Research by the way). Buffering doesn't make much of an I/O speed difference for a single drive, but CPU cycles per byte transferred drop to 1/10th of the amount with buffered I/O (the study used the .NET Framework).You all have confirmed my suspicion that custom low-level I/O is way more complicated than what I need.The reason a typical database doesn't suit my needs is because I need to look up 3D data with continuous levels of detail. Most RDBMS's today have quad tree support for spatial data of several dimensions, but the LOD aspect of my problem isn't well suited to records, and the data will be read-only in production. Also the data objects are really not that big, I just want to read them quickly :-)I will have to go through those systems in your link phresnel; I don't know that any of them will do what I want here, but it looks like good information!I think I have a direction now, but I still appreciate any comments on the matter.   ;Quote:Original post by JonWbecause it is fairly simple to do your own caching (it only adds a few constraints on the sizes of buffers as well as their placement in memory and in the file).Well..."There are only two hard problems in Computer Science:   cache invalidation and naming things."   -- Phil KarltonQuote:Also the data objects are really not that big, I just want to read them quicklyHow big is all of the data?   ;Quote:Original post by Antheus"There are only two hard problems in Computer Science:   cache invalidation and naming things."   -- Phil KarltonYes, that is true, the second problem especially. In my case, I won't be doing much "caching", as I will be using an index to jump to a position in the 3D file and then do a sequential read of large blocks (64KB+) to extract some information from it. Once a block is processed in the sequence, it can be thrown out. As for the size of the entire dataset--maybe a few hundred MB to a couple GB. That's not that big anymore, right? ;);  Another useful technique for fast writing if anyone is curious, is to preallocate the file. A good explanation of the technique from http://www.pcreview.co.uk/forums/thread-1484343-2.php):Quote:> NTFS zeros the file whenever you extend it. If you just use WriteFile()> to extend the file, the same zero'ing occurs. This is for C2 security> requirements, to prevent folks from extending their files over other> peoples files and taking control of the data.>> If you extend a file a little at a time, then for every extension:> Seek to cluster location> Zero out enough space for the addition> Seek back to the start of the write> Write the file data>> (so, 2 seeks for the write)>> Also, if something else has written to the adjoining cluster in the> interim since the last write, then the file will fragment. This happens> all the time in logging applications.>> So, if you pre-allocate the file, you only pay for the zeroing 1 time.> This will prevent any other files from intruding on your nicely sequential> allocation and optimize write speeds. If you only pay the price once,> then the the runtime performance is the same as bypassing the zeroing.> The clusters do not get zero'd until you call SetEndOfFile(). So the> order is:>> CreateFile> SetFilePointer (size of file you intend to create)> SetEndOfFile (commit the file)> SetFilePointer(beginning)> start writing>> No more zeroing will occur unless you overrun the file size allocated.> Also, fragmentation will be minimized. The zeroing happens very, very> fast b/c there is no seeking. A single seek to the start of the file,> then sequential writes so the writes occur at the disk speed.You can use the SetLength() function in .NET for the same purpose.   