What counts as transactions fee in Coase Theorem?
Transaction costs (or fees, though I have always heard it as costs) is anything that complicates or prevents a transaction. This includes the example you are suggesting, but also things like the costs of suing, or if many people own the river and it is hard for them to organize and strike a deal. Maybe the legal system is terrible and people break deals all the time with impunity, etc, etc. 

What you want to say in such cases is that transaction costs are high. Only if something changes that make it harder to strike a deal, it would be appropriate to say that transaction costs increase.