Why is the hit rate so bad until the cache has a certain size?
The "certain size" is known as the "working set" of a piece of code, and the size of the working set varies considerably among different pieces of code.

The question is whether or not a particular item in the cache gets replaced by something else before it can be reused. You only get a decent hit rate if the cache is large enough to hold most of the working set without replacement occurring.