It's all a matter of trust.  Obviously if you are freaking out about this, there is something about him or about your relationship with him that is making you suspicious in this situation.  \nGetting obviously angry with him when there is no proof of wrongdoing will only make him angry and defensive in return, and nothing positive can be accomplished when that happens.  \nExpress to him (calmly) how this made you feel-- if he is at all worth your time and energy, he will totally understand how uncomfortable this must have made you, and be willing to discuss the reasons why.
do i have the right to be mad?