Professor wants to expropriate submitted code for commercial use
You seem to be asking about two different things: (a) is it OK to assign coding assignments based on the instructor's real-world experience, and where the instructor isn't sure how to solve it himself? (b) is it OK for the instructor to use your solution/your code in his company, without your permission?  You should keep them separate, as they're likely to have very different answers.

Is it OK to assign this sort of assignment?  Yes, absolutely!  It sounds like it could potentially be a great project, if designed well.  A course project that's based on a real-world problem?  Sounds wonderful.  The instructor isn't sure what the best way to solve it will be?  Open-ended projects can be great, too, as a way to boost creativity.

Of course, it's the instructors responsibility to design the course project so it will inspire learning and meet the pedagogical goals of the course, but nothing about "based on the instructor's real-world experience" or "the instructor doesn't know how to solve it himself" is incompatible with that (and those could even be beneficial features).  So, your negative reaction to that aspect is unjustified.

Is it OK for the instructor to use your solution in his company, without your permission? Almost certainly not.  There are both ethical and legal concerns.

From an ethical standpoint, this seems like a potential abuse of the instructor's position of power.

From a legal standpoint, this could run into intellectual property issues for the company.  At the university I am most familiar with, the university has an official policy on copyright.  It explicitly states that student coursework remains the student's property: the student retains copyright in essays, code, etc. that they write in their course.

So, look into your university's official policy.  If the university has a similar policy, then the situation is very clear: this is a great assignment, and the instructor has every right to offer this assignment -- but he cannot use your code at his company (without a written signed agreement from you transferring copyright, and it would of course be highly improper for him to expect/demand this as a requirement of the course).

Also, depending upon laws in your jurisdiction, it's possible that this abuse of the instructor's position of power could itself pose legal issues.

In his enthusiasm, your instructor might not have realized these issues or thought through them very carefully.  Don't assume bad faith -- this could well be a situation where the instructor saw a great opportunity for learning, and failed to recognize the issues with this aspect of the assignment.  I'd suggest you start by talking to the instructor in person and very respectfully having a discussion about this: you could start by mentioning that you are not comfortable with him using your code for anything beyond the course, and his statement that he would use solutions makes you uncomfortable.  Have a meeting in person -- an email is too easy to misconstrue.  Assume good faith, share your concerns, and see what he has to say.  It's possible that this might be a misunderstanding or that he might be entirely receptive to your concerns.