Colleague indirectly accusing me of plagiarism? Help?
I worry that you have a misunderstanding about plagiarism and what it means that you had an agreement about things you could use. Normally, plagiarism is pretty obvious on its face. The exception is when there is parallel work by people unknown to each other. 

Even the latter can be complicated in situations where it is fairly clear that the writer should have known about the other work and ignored good research practice. 

But having been given permission to use things doesn't give you permission to use them without citation. You still need to cite the source of anything you use that you didn't develop yourself. 

So, if you did some work with others, and they think they have some claim to originality, so that you obviously know of their work, and they then find a paper of yours that has the work but no citation, it is pretty clearly plagiarism. Doing so accidentally or not understanding the rules is not a defense. You should know the rules and should adhere to them rigorously. 

I don't claim this is what happened here, but it seems that others think it is. Your defense would be to show the citations or to make a claim of parallel, independent, work.  

And they don't owe it to you to tell you before they make a claim. They need to be careful of false claims, of course.