Should I be worried of being accused of cheating?
I'm going to just guess here that there won't be any problem. If there were, you'd likely have been challenged when you left the room. I'll also guess that no one even noticed. 

You seem to have a decent sense of what is allowed. If you are challenged, just tell it like it happened and your thinking about what you did at the time. No one but your instructors can judge, but I think the issue is past you. Probably fine to just relax. 

But note, also, that none of my statements are definitive. Too hard to make firm statements from a distance.