counting (large number of) strings within (very large) text
Question: I've seen a couple variations of the "efficiently search for strings within
file(s)" question on Stackoverflow but not quite like my situation.

  * I've got one text file which contains a relatively large number (>300K) of strings. The vast majority of these strings are multiple words (for ex., "Plessy v. Ferguson", "John Smith", etc.).

  * From there, I need to search through a very large set of text files (a set of legal docs totaling >10GB) and tally the instances of those strings. 

Because of the number of search strings, the strings having multiple words,
and the size of the search target, a lot of the "standard" solutions seem fall
to the wayside.

Some things simplify the problem a little -

  * I don't need sophisticated tokenizing / stemming / etc. (e.g. the only instances I care about are "Plessy v. Ferguson", don't need to worry about "Plessy", "Plessy et. al." etc.)

  * there will be some duplicates (for ex., multiple people named "John Smith"), however, this isn't a very statistically significant issue for this dataset so... if multiple John Smith's get conflated into a single tally, that's ok for now.

  * I only need to count these specific instances; I don't need to return search results

  * 10 instances in 1 file count the same as 1 instance in each of 10 files

Any suggestions for quick / dirty ways to solve this problem?

I've investigated NLTK, Lucene & others but they appear to be overkill for the
problem I'm trying to solve. Should I suck it up and import everything into a
DB? bruteforce grep it 300k times? ;)

My preferred dev tool is Python.

* * *

The docs to be searched are primarily legal docs like this -
<http://www.lawnix.com/cases/plessy-ferguson.html>

The intended results are tallys for how often the case is referenced across
those docs - "Plessey v. Ferguson: 15"


Answer: An easy way to solve this is to build a trie with your queries (simply a
prefix tree, list of nodes with a single character inside), and when you
search through your 10gb file you go through your tree recursively as the text
matches.

This way you prune _a lot_ of options really early on in your search for each
character position in the big file, while still searching your whole solution
space.

Time performance will be very good (as good as a lot of other, more
complicated solutions) and you'll only need enough space to store the tree (a
lot less than the whole array of strings) and a small buffer into the large
file. Definitely a lot better than grepping a db 300k times...

