Why so few Iwatobi Swim Club members?
It is not that they didn't try getting more members, but it's just that no one showed interest in joining the swimming club. In one of the episode, it is shown that the club members put on a show to recruit new members, but fail drastically. 

I don't remember the episode number as of now. I will add the reference when I find the episode. It was like a stage in which each club showcases themselves to get new members from the first years.