Tokenizing a large document
I would guess that you're never going to take another 50% off. You could consider pulling the problem down into C or Haskell or Awk or whatever and then binding that more tightly optimized implementation back up into python; IDK. 

As for more incremental improvements, it depends a bit how the thing is going to be used. 

You flagged the question for multi-threading. That certainly might help, but you'll have to figure out how to divy up the job. Maybe there are natural split-off points that can be identified without actually reading the subject data, in which case any parallelization system will probably work. Maybe you can figure out the size of the subject on disk, and open multiple readers against that file starting at evenly spaced offsets; in this case you'll need to figure out the fence-posting when you combine the results of each of the workers. 

I notice that the function preprocessing just wraps the iterable results up in a list. Is that necessary? If you can avoid ever actualizing the whole 10k-items list in memory that will likely help some.

Similarly, you're reading the whole file (http payload) as a single string. Both file handles and HTTPResponse objects will let you read just a chunk at a time; this would let you handle the subject data like a lazy iterable (similar to the way you're yielding from .finditer()). Of course how much of an advantage this is for you will depend on the underlying implementation, and you'll have to be careful about fence-posting again. This may also be a good way to break off jobs for parallelization.