What is the proper way to validate that a many-to-many relationship will not surpass a certain limit?
SELECT ... FOR UPDATE will not work in this case. It still requires SERIALIZABLE isolation level because what you need to do is to prevent other sessions from inserting rows with the same groupId. And to do this you need to lock a key (what actually SERIALIZABLE level does). What you can try to do?:

check and adjust your indexes (to not lock more rows than you actually need)
revise your code to make your transaction as small as possible
try to emulate locks using separate table (with the list of groupids and locked flag) and updating a row for required groupid before start your transaction and after the end of it.