Video Controller Design
This is why VGA displays didn't happen until long after the Z80. 

Block graphics with sprites, or 1 bit per pixel monochrome graphics, or character ROM based display modes were the order of the day. Sometimes you could switch modes between them.

Using character ROMs, the ASCII character code provided most of the address into ROM, with (scan line mod 10) providing 4 LSBs of address. So you stored ASCII character codes into a small (maybe 2kbyte) RAM, and the video controller hardware read a string of 80 from this RAM, (10 times in succession for 10 successive scan lines) to deliver 80 bytes (640 bits) per line.

Those 80 bytes may come from a 128 byte section of the 2K RAM, to simplify the video addressing. Likewise the 10 scan lines come from a 16 byte section of the character ROM to simplify addressing. With a suitable design you can select a different page of character ROM with 14 or 16 bytes per character for a prettier font (and fewer lines of text on screen!) with only minor changes to the video hardware.

The 6845 video controller was a popular device that could handle this sort of addressing and simple bitmapped graphics; it should still be easy to find a lot of information on it.