What's the psychological basis for "Look at me when I'm talking to you"?
This article may help Basically if you look anywhere but at the speaker you can seem less attentive and looking at them shows you are paying attention to what they have to say

It is also one of a few primary elements in building rapport in those who you have just met and also to maintain rapport especially when listening to a problem the speaker has in a therapeutic or even business relationship.