If capsaicin is insoluble in water, what makes chilli water taste spicy?
You got me curious, so I poked around a bit on this. 

First there is insoluble and then there is insoluble. The Wikipedia article on capsaicin lists its solubility as $\pu{0.0013 g}/\pu{100 mL}$ which is $13$ parts per million. So capsaicin is "relatively insoluble", but not wholly so. Second the Wikipedia article also points out that capsaicin itself is just one of the various capsaicinoid chemicals which give peppers their "hot" taste.  

At the end of the (old) Wikipedia article on the Scoville scale, a taste test for hotness, it states 


  [...] A measurement of one part capsaicin per million corresponds to about 16 Scoville units. [...] 


So, 13 pmm multiplied by 16 Scoville units per ppm is about 200 Scoville units. 

A table on hot sauces and peppers lists the Original TABASCO® brand Pepper Sauce from McIlhenny Company as being 2,500 to 5,000 Scoville units. So 200 Scoville units would be about one tenth strength sauce. That should still be enough to tingle your tongue.