Saying 不是很多人 or 没有很多人 sounds clunky; how can I say "not many people" smoothly?
In fact IMO 不是很多人 and 没有很多人 both sound pretty natural to me, and I think they are the right translation of "not many people". 

"绿腰带不是很多人喜欢佩戴的" sounds a bit odd but mainly that's because you would usually say "不是很多人喜欢佩戴绿腰带" or “喜欢佩戴绿腰带的人不多".

"似乎这里没有很多人会手语" sounds perfectly natural to me.