Should I leave the first authorship of our paper to the student who did the project whereas I solved it?
It would, perhaps, be generous of you to do so, but not necessarily wrong. In math and CS, listing authors alphabetically would probably be appropriate with little or no attention to "first" authorship. 

But an academic being seen as generous is not a bad thing. Too many questions here are just the opposite, demanding first authorship, sometimes properly and sometimes not. But such generosity is more typical in a professor with an established reputation than in a current grad student. 

But a serious consideration would entail looking at whether the student made a significant intellectual contribution to the work. If not, then first authorship is probably overly generous and might set a bad precedent in the mind of that student. 

I'm with your professor on this. Your call. But the advice in comments (BrianH) on setting expectations at the start is good - even essential.