What electronic component of a computer produces (sound) noise?
Yes, components can produce noise. 

Common noise producers are coils and capacitors.
Magnetostriction for coils (i.e. inductors/transformers) and Piezoelectricity for (ceramic) capacitors are the reasons.
These effects cause the component to change shape ever so slightly, and if the applied frequency is in the audio range we can hear it as a buzz.   

This is why you can often hear a high pitch whine in an SMPS power supply (an ATX supply is an SMPS supply)