Character mask VGA display
You are definitely on the right track.

For example, the J. B. Ferguson "Big Board" (CP/M) computer used exactly that technique and technology to produce a 25&times;80 text display. It has 2k of SRAM to hold the characters and 2k of EPROM as a character generator, which could just as well be SRAM too. All of the timing and control is done using LSTTL logic. It uses a clever address mapping technique (requiring just one additional mux chip) that allows the character RAM to be addressed by rows and columns instead of as a linear memory space.

If you want to investigate further, I scanned the documentation some time ago and put it online here.