Changing advisors during coursework
Advisors can do this. (So can students.)  I would ask for some reason (money, competence, personality, etc.)  Not an interrogation or a debate (for now).  Just listen, don't argue and take notes on what he says.  Heck for all you know, his concern may be valid and may change your behavior with a subsequent advisor.

You should also go to the department chairman to discuss your options.

The situation becomes more serious the more time that you have spent with the professor.  If there has been a fair amount of time and you are at least semi-competent, his colleagues may pressure him to just work with you and get you past the goal line.  Every situation depends on the particulars (or even how the department behaves).  But alternately, if you are a "problem child", the professors may back up their colleague.  

I would not expect some Roman judge standard of fairness, but I would certainly look at your options and talk to department chair, alternate advisors, and maybe even the grad school.  If you are a decent candidate and things are early, it may be simple to just move to an alternate advisor.