How can I tell a professor that I don't want to join his group? (science major)
It sounds like you have not made any sort of commitment, so if that is the case then there should be absolutely no hard feelings by simply writing a polite email saying that you have decided not to do research with him.

If you wish to elucidate the reason, just say you don't feel that you'd fit in well with the group. He'll understand. You should not feel obligated to come up with an elaborate explanation.

It's hard to tell without more information on the course you speak of. If it's an ad-hoc, informal course taught by the professor himself then I can see how the investment of time may only be worthwhile for his own students. If it's a proper course that you pay fees for and earn credits, then perhaps it is reasonable to ask if you can still do it.