Is it normal for an instructor to charge students to complete copyrighted homework?
I think that, at best, there is a conflict of interest for an instructor to force students to use a web service that charges money of which s/he gets a cut.

That said, are you sure that that's how it is? For textbooks -- where the same conflict of interest exists -- it is common practice at many universities that if a professor uses her own textbook for a class, that that part of the royalties that the professor receives through his own students' purchases actually goes into a university fund outside the professor's control. The same may in fact be the case at your university -- the professor may simply never have talked about the details in class.