When is a paper said to be refereed?
First, note that there is a difference between refereed in a peer-review journal and refereed in a conference proceedings. It sounds like your case is the latter. 

In both cases you are correct in saying it received peer review (if it did), but the moniker of 'refereed', in my mind, suggests the work could be rejected. 

You note that you received 'reviews' but it isn't clear whether these are non-binding 'comments' or part of a dialog ending with accept/reject. If they were indeed just comments, and there was no possibility for rejection, I would not say it was 'refereed'.  Like in sports, referees make calls, not suggestions.