Commonly used test image sets
Segmentation : Berkeley Segmentation Dataset and Benchmark

Stereo : Middlebury Stereo Datasets

Shape from specular flow - Shape from specular flow dataset

Also, check out this link : http://clickdamage.com/sourcecode/cv_datasets.php