What is the role of capacitor C3?
Wouter's comment is correct : this is a Colpitts oscillator, where the transistor operates in common-base configuration. So C3 keeps the base voltage grounded at AC, though its DC value is positive to bias the transistor. An electrolytic capacitor will be largely inductive by these frequencies, so if you use one it should be in parallel with a smaller ceramic capacitor, as you found.

Because a common-base amplifier is non-inverting with high voltage gain, capacitor C2 provides positive feedback from the collector (output terminal) to the emitter (input terminal) to sustain the oscillation. 

(I have to say C2 looks unusually large in relation to C1, this may lower the Q of the oscillator, making it less stable than ideal)