Is the masking effect of sound related to action potentials or to mechanical aspects of hearing?
Short answer
Both - masking occurs through neural effects, as well as via mechanical effects in the inner ear. 

Background
First off, there are basically three types of masking, namely simultaneous, forward and backward masking.

Simultaneous masking experiments are the most basic and likely the most widely used. The most basic type of experiment is listening to tonal stimuli in static (white) noise. In this case, the inner ear is busy processing noise and acoustic probe stimuli (tones) are not perceived by the subject, while loud probe stimuli are noticeable. A more relevant test for everyday functioning is the speech-in-noise test, where speech stimuli are presented in noise. It appears that hearing loss causes folks to have impaired hearing in noise. The amount of impairment in speech understanding in noise exceeds by far the simple decrease in hearing sensitivity. Anyway, I'm drifting off.

Forward masking experiments present a masker before the probe.

Backward masking experiments present the probe before the masker. 

To take a step back - the auditory system consists of the peripheral end organ, namely the hair cells in the inner ear (Fig. 1). Those hair cells translate acoustic physical stimuli into electrical currents. These are currents are fed to the spiral ganglion cells in the inner ear. Their axons bundle up in the auditory nerve which innervates the brain stem and ultimately the auditory cortex in the brain. 

Now - forward masking experiments are generally regarded to target the peripheral structures, while backward masking experiments target the central structures at a higher level.

Peripheral masking effects occur at the level of 


the basilar membrane (e.g. Zwicker (1977)
the auditory nerve (e.g. Stronks et al. (2012)


The basilar membrane is the structure that mediates the (traveling) wave that sounds generate and that traveling wave is the very basis of the frequency analysis in the cochlea, akin to a place-dependent Fourier transform. It is the basilar membrane that separates frequencies from high at the base to low frequencies in the apex due to its physical properties. Now, when the basilar membrane is busy moving in response to a masker presented at 1 kHz at 100 dB SPL, a probe tone of 1 kHz presented at hearing threshold (0 dB HL) will be totally swamped, simply because the amplitude of basilar membrane movement will be changed by next to nothing. 

The auditory nerve is also an important neural site where masking occurs (e.g., Stronks (2010). This has to do with the fact that neurons fire action potentials. After a neuron has fired an action potential, there is an absolute refractory period that takes several ms to resolve. In that time, ion channels previously engaged in firing are re-opened and up and ready for the next action potential. Within the absolute refractory period, no stimulus, whatever its strength, can evoke another response in that neuron. Following that, a relative refractory period follows in which its firing threshold is increased, but strong enough stimuli can evoke a response. If a neuron is busy firing after chronic stimulation, it gets tired and a state of adaptation follows. Further stimulation results in fatigue, in which the neuron gets tired and has exhausted its resources to fire. All these processes can result in masking of simultaneously presented stimuli (simultaneous masking), but notably also in stimuli presented after the masker has subsided (forward masking). 

Now, all these processes will not affect backward masking, where a short probe stimulus is swamped by a following masker (e.g., Oxenham &amp; Moore (1995). This is caused by more central, higher-level processes that include attention processes, i.e.,


the brain


In other words, the following masker distracts the attention of the subject that in turn fails to notice the probe. It's thought to depend on central processes as peripheral processes can't be 'fooled' and faithfully transmit the stimulus to the brain, whatever comes next.   


Fig. 1. The inner ear (cochlea) features hair cells that are actiovated by sound waves traveling through the basilar membrane. source: Modesto Junior College

References
- Oxenham &amp; Moore, JASA (1995): 98; 1921
- Stronks, PhD thesis, 2010, University of Utrecht
- Stronks et al., Otol Neurotol (2012); 33: 35-42
- Zwicker, JASA (1977); 61: 1031