Is it ethical to ride with races without being part of them?
Unethical, because you're imposing costs on the organisers anyway. Apart from the obvious "if you get hurt they're going to help you", they almost always face costs per participant and often have a limit on numbers. Violate those limits and the organisers will have problems.

You're also causing wear and tear on the track, equipment and marshals. You can't pretend that away by saying "other people do it too"... yes, and they paid for the privilege. You're stealing it. Not really ethical IMO.

This is another case where your exact marginal impact is low, but at some point one more increment like that will get the race cancelled or refused permission to do it again.

The other thing is that if you are involved in an incident and are deemed to be at fault, you will not be insured. The race insurance won't cover you (obviously), but whatever insurance you have almost certainly has a an illegal activity exclusion. They'd most likely leave it to you to show that what you were doing was lawful. But since most races take place on temporarily closed areas, that would be hard. They close the area specifically to people like you. For exactly that reason.