In our bioinformatics application we have large files (up to 900MB, typically 100MB). These files contain a clear text representation of a genome, and are essentially a 1 line file with a sequence of characters.Data is referenced by location, for example chromosome 7 starts at position 1 and ends at position 158937463. We typically extract a small section of around 400 characters, eg: from position 4,120,000 4,120,400.We have a utility written in Ruby for this purpose: https://github.com/sfcarroll/bio-fasta-read and it works by reading through the file from the beginning. We do these reads many times and they are slowing our application. I was wondering what options are available for caching? The source data will never change, but the ranges will frequently change. We are running Ubuntu Server 14 x64 on a machine with 128GB of RAM.Is there some OS-level way of increasing performance? Perhaps loading the entire file into memory or somehow caching the requests?EDITI should add if there are options such as somehow assigning more memory to the file cache that would help, pointers would be appreciated. We can consider a dedicated server for these file reads if it would need to be tuned in a specific way.EDIT 2 We are running a Xeon E5-1650 Hexa-Core CPU and dual SSD's (that could be raided), along with 128GB RAM.
Increasing performance of clear text file reads
linux;ubuntu;performance;cache
The linux kernel does cache management automatically. Everthing that is loaded into RAM, stays there until another process needs RAM and there is no more available. So in the linux kernel RAM should always be full. You system has 128GB of RAM, thats more than enough for a 100-1000MB file.To load a huge file into RAM just cat it:cat huge_file > /dev/null 2>&1All output is sent to /dev/null, but for that it must pass the systems RAM. While doing this you can watch how Cached in /proc/meminfo increases.When cat ends, execute the Ruby application. The Ruby application now reads the cached version of the huge file.