Trouble designing an ALU
There is nothing wrong with the way you suggested. It may cost a few more gates than a highly optimised solution, but gates are (usually) cheap and designer's time is not.
It is far more important to design something correct, clear and understandable, and your approach can achieve that.
Plus, any synthesis tool will aggressively optimise whatever you produce anyway.