What's the purpose of "over G" penalties in Red Bull Air Race?
It's a safety rule to protect the limits of the airframe and the pilot.

See the video (also on youtube) from this Red Bull Air Race page which explains the rules. The G-limit is discussed from 3:30.

The following is a quote from the relevant section of the video.


  Maximum load factor, that is how much G's is the pilot pulling. That has
  issues [...] the structure of the
  airplane is designed to withstand a certain amount of G. We need to
  ascertain, to check, that the pilot isn't exceeding that amount of G,
  so we monitor that very carefully. A thousand times a second we're
  monitoring the G inputs that the pilot is putting on the airplane. So
  if you exceed the maximum G's then DNF (Did Not Finish)