Speeding up python load times for read only dictionary?
I'm not sure there is a right answer here since there are some tradeoff, BUT.
two options come to mind:
1st. consider using panads.DataFrame for the data-stucture.
It will allow serialization/deserialization to many formats (I believe CSV should be the fastest but would give SQL a try). as for query time, it should be much faster than a dict for the complex queries.
2nd.
key value store, such as MongoDB that has map-reduce and other fancy query capilites, in this case the data is always available without loading times.
I currently am working on a bioinformatics project that currently involves a dictionary corresponding to about 10million unique keys, which each return a subset of categorical strings.
I currently use unpickle a dictionary object, but my main issue is that unpickling takes a very long time. I also need to iterate through a file, generating a set of keys(~200) for each row, lookup the keys, appending the list to a list-of-lists, and then subsequently flattening the list to generate a counter object of value frequencies for each row, and I have heard that a SQL database like structure would end up trading load times for lookup times.
The file that has keys typically contain about 100k rows and so this was my best solution, however it seems like even on faster pcs with increased ram, num of cores, and NVME storage that the time spent on loading the database is extremely slow.
I was wondering what direction (different database structure, alternatives to pickle such as shelves or mashall, parallelizing the code with multiprocess) would provide an overall speed up (either through faster loading times, faster lookup, or both) to my code?
Specifically: Need a create databases of the format key -> (DNA sub-sequence) : value ->[A,B,C,Y,Z] on the order of 1e6/1e7 entries.
When used, this database is loaded, and then given a query file (1e6 DNA sequences to query), perform a lookup of all the sub sequences in each sequence do the following.
For each query:

slice the sequence into subsequences.
Lookup each subsequence and return the list of categoricals for each subsequence
Aggregate lists using collections.Counter

I was wondering how to either:

Speed up the loading time of the database, either through a better data structure, or some optimization
Generally improve the speed of the run itself (querying subsequences)