How to limit number of contacts in a group?
Just to be sure: civi doesn't have any limit on the number of contacts per group, it's a business case you have that introduces that constraint, right?

I didn't find a safe way to prevent an operation (I'm not sure it's a good idea, it might introduce lots of problems later on, the best IMO is to fail the query). 

One workaround to get that is to introduce an invalid value in the pre hook (eg set contact_id or group_id to null, the query will fail and it will prevent inserting.