Should I remind my potential PhD advisor to know his feedback regarding my solutions to a problem set given by him?
Yes, absolutely. If it's been more than a week, it's definitely appropriate to send a polite reminder. He probably gets a lot of e-mails and may have just missed yours.

In general, I've found it's best to follow up after a week if there's any chance that the other person missed or forgot about the e-mail. If he's aware that he still needs to respond, no harm done. If he's not aware, then it's a good thing you're letting him know.