Why don't trigonal S and P compounds undergo inversion at room temperature?
Azane all by itself has an inversion barrier of about $6\ \mathrm{kcal/mol}$, which is  low compared to phosphine's approximately $30\ \mathrm{kcal/mol}$ barrier. Pyramidal nitrogen inversion requires a planar transition state. One can use a classical theory to model the thermal rate and we could write $$k \propto \mathrm e^{-E_\mathrm a/(RT)} $$ and using models like these we would predict that ammonia at $300\ \mathrm K$ would have a rate on the order of $10^8\ \mathrm{s^{-1}}$, instead what is observed is something on the order of $10^{10}\ \mathrm{s^{-1}}$

Assuming the potential energy surface has two minima equal in energy and a higher energy TS, two routes are possible: a thermal process going over the maxima or quantum-mechanical tunneling. Determining a tunneling frequency requires writing down vibrational wave functions, finding their linear combination and overlap. If we were to do all this for a particular vibration, we'd see that the tunneling frequency decreases exponentially with a dependence on increasing $\mu$, thickness of the barrier/shape, and the barrier height. As an example, $\ce{ND3}$ has approximately an order of magnitude slower rate of inversion than $\ce{NH3}$ at a sufficiently low temperature – this has been attributed to a decrease in its tunneling frequency.

But why the difference in the energy barriers and corresponding rates depending on the atomic position in the periodic table? Perhaps the easiest/quickest explanation rests on considering what the TS requires for interconversion. Ideally the classical conversion requires passing through a trigonal planar structure with bond angles on the order of 120 degrees. If you consider the series of azane, phosphane, arsane then you'll see bond angles of 108, 94, 92 degrees respectively. In the absence of other effects, the barrier goes from smaller to larger. Indeed, asane-like molecules were the first of this series resolved, followed by phosphane-like. To the best of my knowledge the fastest have yet to be resolved, because rates on the order of $10^{-5}\ \mathrm{s^{-1}}$ are required at RT for a decent shot at isolating them. Naturally, this isn't the whole story, as I've previously alluded to above. Let me know if you would like to know more.