Psychoacoustic Masking Curve
Psycho acoustic masking is a VERY complicated subject. The masking thresholds (energy required to have one sound be audible in the presence of another) is a very complex function of 


Spectral content and overlap
Spatial presentation (there are many books just on binaural masking alone)
Time shapes of the waveforms
Time alignment and envelopes of the wave forms


Even the more advanced audio codecs (such HE AAC) use very simplified models because a more accurate one is very complicated. 

The best choice really depends on your application and requirements, monoaural masking of simultaneous sinusoids is probably the easiest case. You may find some helpful stuff here https://engineering.purdue.edu/~malcolm/interval/1998-010/