It could mean that he finds you attractive and it makes him nervous especially if he only does it with you. It could also mean that he doesn't like you, he's not interested in you, he's nervous to make a move, he's naturally quiet or he might not like the people you hang around with.