Understanding the circuit diagrams for white noise generators
1) What is the purpose of R1 and its connection to ground?


1N759 is a zener diode (12V).  R1 lets current flow through the zener diode (connection in reverse direction is the typical application for zener diods)


  2) What is the purpose of the 5V output?


Power supply for the amplifiers


  3) What is the purpose of the capacitors here at all? What are they serving to do? Especially C3. It doesn't even look like outputted noise goes near C3. 


C1 and C2 are high-pass filters, while C3 stabilizes the power supply for the amplifiers.