Where to start when considering making a GPU?
That's kinda like going to your collage final exam for science class and having this as your question:  Describe the universe.  Be brief, yet concise.  There is no way possible to answer that one in any practical way-- so I'll answer a different question.  

What are the kinds of things I need to know before attempting to design a GPU?

In a rough chronological order, they are:


Either VHDL or Verilog.
FPGA's (useful area to play with writing digital logic).
Basic data-path stuff, like FIFO's.
Bus interfaces, like PCIe and DDR2/3 interfacing
Binary implementations of math functions, including floating point, etc.
CPU design.
Video interfacing standards.
High speed analog stuff (the analog side of high speed digital)
PLL's and other semi-advanced clocking stuff.
PCB design of high speed circuits.
Low voltage, high current DC/DC converter design.
Lots and lots of software stuff.
And finally, ASIC or other custom chip type design.


I will also dare say that you won't be making this kind of thing out of TTL logic chips.  I doubt that you could get a reasonable DDR2/3 memory interface working with normal TTL chips.  Using a big FPGA would be much easier (but not easy).

Going up to step 6 will probably be "good enough to quench your intellectual thirst".  That could also be done within a reasonable amount of time-- about a year-- to set as a short-ish term goal.

EDIT:  If all you want to do is spit out a video signal then it's relatively easy.  It is, in essence, a chunk of memory that is shifted out to a display at 60-ish Hz.  The devil's in the details, but here's a rough outline of how to do this:  

Start with some dual port RAM.  It doesn't have to be true dual port ram, just some RAM that a CPU can read/write and that your video circuit can read.  The size and speed of this RAM will depend on what kind of display you're driving.  I personally would use DDR2 SDRAM connected up to the memory interface of a Xilinx Spartan-6 FPGA.  Their "memory interface generator" core (MIG) makes it easy to turn this into a dual-port RAM.  

Next, design a circuit that will control how this RAM is read and spit this data out a simple bus.  Normally you just read the RAM sequentially.  The "simple bus" really is just that.  It's some bits with the pixel value on it-- and that's it.  This circuit will need to do two more things:  it will have to go back to the beginning of RAM every video frame and it will have to "pause" the output during the horizontal/vertical retrace periods.

Thirdly:  make a circuit that will output the video control signals (HSync, Vsync, etc.) as well as tell the previous circuit when to pause and restart.  These circuits are actually fairly easy to do.  Finding the appropriate video standard is harder, imho.

And Finally:  Connect the control signals and video pixel data bus to "something".  That could be a small color LCD.  It could be to a video DAC for outputting a VGA compatible signal.  There are NTSC/PAL encoders that would take these signals.  Etc.

If the resolution is really small you might get away with using the internal RAM of the FPGA instead of an external DDR2 SDRAM.  I should warn you that if DDR2 SDRAM is used then you'll probably require a FIFO and some other stuff-- but that too isn't terribly difficult.  But with DDR2 SDRAM you can support fairly high resolution displays.  You can also find FPGA development boards with integrated VGA DAC's and other forms of video outputs.