Why is my VHDL clock signal so far off from what I thought it would be?
50MHz/4.16666MHz = 12.

You are counting to 6 twice, not 5 twice as you think you are.
The classic off-by-one error :)

If it seems muddled, try simulating your code, and that ought to clear things up.