Suspected a person had an affair with the professor
As you suspect, you should just let it go. I think it is indeed reasonable for the situation to prey on your mind, but it is beyond your power to really do anything positive or negative, and, as you say, any attempt to do so would probably just create unpleasantness without being effective.

The potential "victims" of the situation (apart from the student, we may argue... even if it is not clear in the moment) are other students, if any of them get worse grades due to the situation, or if they come to suspect that there's some hanky-panky going on, which would be suspected to influence grades. At the very least, the principle that anything distracting from the immediate goals of a class is undesirable... means that an affair between a student and teacher is bad in that sense.

But, for you, just let it go, just as for most purposes we must just "let go" our disquiet with bad news on television. Yes, _if_possible_, we should not fail to act to improve things. But our plausible scope of action is not global, and maybe not even too much about every "local" thing.