It probably means that said person affects the way you feel, good or bad, whether you like it or not. Telling yourself not to think about him/her would be counter-intuitive. ... So if you do not want to think of a specific person, think of something else instead.