NoSQL Databases Performance I have been engineering and developing a NoSQL database engine (using C#.Net) for .Net technologies. Similar to the available NoSQL database, I store all documents in JSON format, and I keep all related document records (i.e. Album1, Album2, Album3) in a single file. 

I have been testing the solution to see its actual performance in a real world scenario, and using Visual Studio Unit Testing framework, my solution has managed to query and search 2,000,000 documents in approximately 12 seconds. 

My first question is, how good is that performance if it is a good result at all?

Secondly, since all records are eventually saved in files, I have implemented a singleton design pattern that buffers up all physical documents in memory in order to prevent the need for concurrent file processing. At 2,000,000 and assuming there are 10 document categories (Album, Genre, Artists,...) this will consume more than 8GB of RAM which is not good. On the other hand, if I disable the buffering and make all queries dependent on searching the physical files, assuming that I get 100 concurrent request and each take 12 seconds to complete, these 100 request may take up to 1,200 seconds to complete which is I think is terrible. 

The second question is, how could this be optimized? I mean, a NoSQL database, unlike a SQL database, is meant to preserve massive data, and such massive data cannot be fully buffered in RAM or be searched over and over on disk. In theory, how should this be implemented?
100,000-ish records per second doesn't sound too bad as a headline number. Without context, however, it meaningless. What's the spec on the storage array? Are you getting even close to the manufacturer's IOPS and bandwidth numbers? How does it compare to a disk performance testing tool's reported values? When profiling the application under heavy load, where is the time taken? Is the system IO or CPU bound?

Some top-end servers do have ridiculous amounts of memory. 64TB is the largest I've seen. But you're right, holding absolutely everything in memory won't work in the general case. There is the concept of the working set. These are the rows that are being read and written by current users. The system holds these in memory and synchronizes them to disk from time to time. When users move on to new work the system makes room by purging the least recently used data from RAM.

For finding specific records you will need indexes. These trade additional disk and pre-processing time for reduced lookup time. There are many different indexing algorithms, each with benefits and drawbacks.

For large OLAP work it might be that you simply must read all of the data off disk to perform the query. It may be possible to pre-process to get summary values (cubes and similar).

This is a huge undertaking. Whole careers are spent trying to get this right. That you have a working product at all is impressive.