Hard to think when hearing audible speech
There are many things at work here. A lot of your sensitivity to human voice can be attributed to the tonotopic organization of the basilar membrane. Check out this image of the cochlea and notice how its sensitivity to different frequencies does not decrease linearly over space https://media1.britannica.com/eb-media/98/14298-004-99934987.jpg , this entails that we are most sensitive to frequencies in the 2-4kHZ range, which significantly overlaps with the frequency range of human speech https://users.cs.cf.ac.uk/Dave.Marshall/Multimedia/node271.html .

There is also evidence for using a network of domain specific mental processes to do complex tasks like navigating a room. Specifically, it seems that linguistic processes make spacial navigation tasks easier(https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3572715/ ; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2745626/) --if this is true we could predict that overloading someone's linguistic capabilities by, say, speaking to them, would make spatial navigation tasks harder, which could explain how some stimuli can be "less distracting" than others within different contexts.