Procedure to follow when added to conference abstract without permission
This kind of thing happens sometimes. It is reasonable to request that you be removed. In many cases, this will be possible while still keeping the abstract. It comes down to how far in advance the conference material is printed/published online.
The organizer mentioned on the conference website can handle this request or forward it to the poor grad student who manages the book of abstracts.

There are a few variables here. If you request the removal of your name directly from the shady collaborators vs. from the conference organizers themselves, it might be helpful in preserving a good-ish relationship with the collaborators. It might also not work if they don't comply.

If you request the removal from the organizers before abstracts have been selected, this might affect the success of that abstract. However, only if the organizers are also the program committee, aka small team.

Ime, this shouldn't affect the group's reputation in the future. Fields probably vary.