Should I argue with the conference chairs in this situation？
If you submitted a month ago and already got rejected, was the new venue truly a "conference" or another workshop?  And, as @Pete L. Clark said, was it published before or only submitted and rejected?  

If published in a workshop (or accepted for publication), substantial portions can't be reused in another workshop.  Portions can be reused in a conference IMHO, but as @Pete said, you have to disclose and cite.  If not, it's self-plagiarism.

If the new venue is a workshop, then it can't be reused that much unless the earlier one was rejected.  In which case the reviewer shouldn't hold the earlier publication against you, unless they meant to say "I saw this before and it was rejected because it had flaws, and now I'm seeing it again".