Should I be concerned about the rank of the journal/conference when asked to be a reviewer?
"Concerned" is not quite the word I would use, but it seems very reasonable to take the quality of the publication venue into account when deciding whether to accept the referee request.  

Most academics I know hit a certain threshold age / level of seniority after which they get more referee requests than they can reasonably take on [here I should say that in my field, mathematics, refereeing one good paper can be a lot of work -- it is no exaggeration to say that doing so may require putting one's own research on hold for several weeks].  So they have to pick and choose which requests to take.  They pick papers through a combination of personal interest, sense of their relative expertise (if you feel that you are one of the only people in the world qualified to competently referee a paper in a reasonable amount of time, it's pretty hard to say no) and estimations of quality of the author/editor/venue.  

Based on what you said, considerations like the above seem to point in the direction of your declining to referee the paper.  If you are so junior that you have almost no refereeing experience at all, then you might want to take it anyway...but then again you still might not.  I would encourage you to discuss this with your thesis advisor (if you are a graduate student) or with a faculty mentor.  You want someone with intellectual and sociological expertise in the field to weigh in on whether this would be a good use of your time.