How to professionally quit a student job?
It sounds to me like you don't exactly need to quit. Remind him of the conditions under which you took the job, and ask him to stick to the agreement. If he can't do so because his requirements have changed, then that's his choice. You might say something like this:


  As you recall, when I accepted this job, it was under the condition
  that I would only work 5 hours per week so it wouldn't interfere with
  working on my thesis. Because of your workshop, I have been working
  far more time than we agreed. I'm concerned that you need more support
  than the 5 hours a week I am able to provide. What do you think?


Note that I said "5 hours per week" twice for emphasis. Give him a chance to respond. At this point, if he promises to stick to 5 hours per week in future, and allows you to take the next few weeks off to catch up on your thesis, then perhaps you might give him one more chance.

However, if he indicates that he needs you to work more hours, or be available whenever he has deadlines, then you might say something like this:


  Unfortunately, I can't work more than 5 hours per week, and it sounds
  like that just won't be enough. Perhaps the best way to solve this is
  turn the project over to someone else. I'll do what I can to ensure a
  smooth transition.


Instead of "quitting", you're working with him to solve his requirements. If it turns out that the only way to meet those requirements is for you to hand off the work to someone else, then that's his choice.

As for the time you worked for free at the start of the contract, I think it is going to be difficult to get paid for that without burning bridges. Of course, you are entitled to be paid, and you could pursue that legally, but you may not want to. In future, remember never to work without a contract.