Image Processing - Counting nuclei I am trying to create a program that can count the number of nuclei in such an image:



What I've already done is the following, step by step:


Apply an Alternating Sequential Filter (closing and opening the image with gradually bigger structuring elements)
Apply a distance transform
Apply watershed segmentation using the distance transformed image to detect minima


Which yields the following result (where each color represents a new nucleus counted):



As we can see, there are lots of imperfections, specifically, overcounted nuclei. I'd say that the reason for that problem is the way I imposed minima for the Watershed Transform (using the distance transform), but I really have no other ideas for imposing minima in that case.

As the Distance Transform generates minima based on the roundness of objects, I would like to know a better alternative to round up the nuclei than the Alternating Sequential Filter (looking at the image above, we can infer that most of the "overcounts" came from the less rounded nuclei). I would also like to know better ways to impose minima for the Watershed Transform.
There are numerous articles on how to handle the oversegmentation problem of watershed, but I think you should read Robust Cell Image Segmentation Methods (scientific article from 2004 by Bengtsson et al).

It covers various methods for segmenting cell images and includes real-world examples that show how to handle oversegmentation from watershed on fluorescence microscopy images similar to yours (it also has examples for bright-field images and confocal microscopy images). It uses seeds from the distance transform, similar to your approach, and merges regions with weak borders. The article reads well and the concepts are pretty straight-forward to implement in Matlab.

For an even more current approach, you can read A Decomposition Scheme for 3D Fuzzy Objects Based on Fuzzy Distance Information by Svensson. It uses a similar method as in Bengtsson et al, but works on the fuzzy distance transform which gives a better density representation for the objects used in the article.