Can I ask a reference letter from my Operating Systems professor while applying to Machine Learning programme?
A good recommendation is one that highlights your potential as a researcher and your related technical and interpersonal skills. If the mentioned professor knows you and can vouch for your potential in these areas, the recommendation will be beneficial. Recommendations are not focused on what courses the professor teaches. A good recommendation can even come from a non-CS professor (but preferably in a related field like EE) whom you have worked with and knows you personally.