I'm not interested in working with this potential advisor. What to tell him? Or should I reconsider?
Given that the supervisor has more experience than you, it may be best to follow his advice. Don't let ego get in the way of improving your work. 

I wonder if your second interaction as described might be the result of his feeling that you were rejecting his advice without considering it. 

On the other hand, it is good to have an advisor you are comfortable with. Whether it is possible or dangerous to change supervisors depends on the particulars of your institution. Since you say "potential supervisor" it would be a good time to evaluate your options. 

But in general, once you have an advisor, it is good to take their advice, for strategic reasons if no other. You want to finish successfully in a reasonable amount of time with good recommendations. Fighting with your supervisor achieves none of that.