The Mother Dough Blues
The sour dough (including the 'mother') should be considered spoiled when:


It smells very bad (like voimit or feces). The natural sour dough smell may be not very pleasant to some, but spoiled dough smells really afwul
It grows hair.  That means mold. And even if it is visible only on the surface, whole dough is spoiled.
It gets strange colour. Generally the dough should be white/gray/brownish, maybe with some darker spots when whole grain flour is used. When it becomes black, red or blue – it is bad.
It doesn't work. Good sour dough will raise when fed and put it warm place. When it is spoiled it usually won't work any more. Good bacteria and yeast are usually mostly dead when bad bacteria or mold took over the dough.