Is it OK to let somebody know I refereed their paper?
but refrained myself because peer review is supposed to be anonymous.


Continue to refrain yourself. You can certainly say that you are familiar with the work of somebody else -- published articles can be read by anyone, after all -- but openly declaring that you were one of the reviewers would not add anything to the discussion and could put you in an awkward situation, if the authors haven't actually liked your review.

I work in a narrow field where almost all of the authors know each other and where it is not so difficult to guess who the reviewers are, also because journals ask the authors to suggest possible reviewers. These authors collaborate quite frequently, but I've never heard of anyone disclosing their role.