How is it possible to drive VGA displays at such high pixel clock frequencies?
There are two main reasons you are finding this challenging.

First, you are using older and more discrete (lower scale integration) parts than would have been used to do this in the era of VGA.

But next, you are using them in an atypical way.  Specifically, your approach is not pipelined which means that you are having to add up multiple delays when determining your interval, and thus rate.

In contrast, synchronous digital designs which attempt to achieve speed try to do as little as possible between registers.  

While the details would probably differ a little, crudely speaking it would work something like this:


You increment or reset the address, then that goes in a register.
You latch the address into the synchronous memory
You latch the output of the synchronous memory
You latch this into the address of the synchronous character generator
You latch the output of the character generator into the output register
apply the palette lookup...
into the synchronous DAC...


When you break a task down like this, you only get one combinatorial delay plus some propagation delay and register setup and hold times needing to fit between clocks.   

A design built this way will take many clocks to produce an output - the latency will actually be higher than a purely combinatorial design.   But it produces a new correct output on each cycle of a much faster clock.  

And hey, it's video, it doesn't really matter if the CRT is drawing a dozen pixels behind the pixel counter - you of course take that into account in the timing of the sync signals so that they are correct compared to when the data actually comes out of the DAC.  

In practice, almost all complex digital systems work this way, as it's a great idea - right up until a pipelined CPU hits a dependency on an earlier computational result or a conditional branch...   Then things get interesting, as they'd talk about in the next lecture of a digital systems class - but fortunately your VGA situation is a lot simpler, especially if you don't yet worry about tearing effects if the character buffer changes while the screen is being drawn.

As a practical matter if you want to build this, do it in an FPGA.  That will pretty much force synchronous memories on you if you use internal ones, or synchronous IO registers if you use external memory.  You'll get a lot of nudging towards a proper design, the fabric itself will be faster than your discrete parts, and of course if you make a mistake you only need twiddle your thumbs while it recompiles rather than spend a long day re-wiring.