Fastest random number generator I just read High performance physical random number generator by K.H. Tsoi, K.H. Leung and P.H.W. Leong. It proposes a physical RNG of speed 400 Mbps. Is this the fastest random number generator at this moment?
The problem with questions that ask for “the fastest” is, that such questions always raise the counter-question: compared to what exactly? Also, your question doesn’t specify if you mean cryptographically secure physical random number generators, or any physical random number generator. 

Anyway… 400 Mbps doesn’t really come anywhere near the word “fastest”.

If you take a look at “An optical ultrafast random bit generator”, which was published in Nature Photonics (Volume 4, Issue 1, pp. 58–61) by Kanter et al in 2010, you’ll see that their physical random bit generator operated at a rate of 300 gigabits per second. I’ld say that somewhat fits the word “faster”.

Besides that, you’ll most certainly also will want to check out “Real time demonstration of high bitrate quantum random number generation with coherent laser light”, published by Symul at al (ANU, Appl. Phys,. Lett. 98, 231103) in 2011. 
In short: researchers at the Australian National University are generating true random numbers from a physical quantum source. 

Currently (meaning: at the time of writing this answer), their Quantum Random Number Generator seems to be the best fit for the word “fastest”. 

To learn more about the QRNG, you could either start by reading this ScienceDaily article,  or you can skip that and directly visit the “ANU Quantum Optics – Quantum Random Number Generator” webpage.

An aside: as others already hinted  at in their comments ⁽¹⁾ ⁽²⁾, even simpler RNGs already tend to break the 400 Mbps mark.