What role does the capacitor play in an AM demodulating circuit?
The capacitor is not required for the demodulator to work. However it does improve its operation is two ways:-


By charging up to the peak signal amplitude and holding most of that charge between peaks, it increases the audio output voltage.






Without any capacitance after the diode, the average output voltage of a half wave rectifier is Vpeak*0.318. With a suitably sized capacitor the audio output voltage increases closer to the peak value. The result is a stronger audio signal, potentially as much as 10*log(3.182) = 10dB louder. 

In practice audio circuits have some capacitance of their own, which will affect the output to varying degrees depending on the rf frequency and complex impedance of the load. For example a magnetic headphone (commonly used in 'crystal' sets) has high inductance which resonates with stray capacitance at some unspecified frequency. An audio amplifier may be connected through a signal cable whose capacitance depends on length, and the amp's input capacitance may vary depending on the volume control setting. Having a parallel capacitor in the detector circuit reduces the effect of varying load capacitance and inductance, making the audio signal amplitude more stable and predictable. 

However if the capacitance is too high then it will fill in the gaps between peaks enough to distort the troughs in the lower part of the audio waveform and reduce the amplitude at higher frequencies. To faithfully reproduce the original modulation there needs to be a balance between filter capacitance and load impedance.


The capacitor filters out the rf frequency.


You can't hear rf frequencies but your amplifier might. Without low pass filtering the rf component of the signal is higher than the audio component. If the amplifier doesn't filter this out internally then its power consumption will be increased and it could produce distortion and spurious frequencies (which could be audible). Effective power output will also be reduced because the amp will overload at a lower audio signal level.    

Even if the amp can handle this rf leakage without distortion, you now have high power rf signals in the audio circuit, which could feed back into sensitive rf stages to cause distortion or even oscillation. Since only a small amount of capacitance couples rf signals quite well, the amount of feedback could change depending on the position of speaker or headphone wires or the operator's body.