Detecting Individual Photos on a Scanned Page I'm working with a collection of old photographs. They were scanned on brown paper in various configurations. Some scans have a single photo, others have three or four. Some of the photos are mounted on white cards, others are not:





I developed an algorithm to find the individual photographs on each scan, which is described here. The gist is to:


Find the median color (independently for each RGB channel, 0-255)
Blur the image.
Binarize according to whether each pixel's RGB values have an RMSE of &lt;20 from the median.


This leaves a mostly-black image with a few large white rectangles on it, like so:



To find the rectangle coordinates, I use the following procedure:


Pick a random white pixel, (x, y) (statistically, this is likely to be in a photo)
Call this a 1×1 rectangle.
Extend the rectangle out in all directions, so long as you keep adding new white pixels.
If this rectangle is larger than 100×100, record it as a photo. (The originals are much larger than the photos I've uploaded for this question.)
Color the rectangle black.
If &lt;90% of the image is black, go back to step 1.


This has worked on most (90+%) of the scans that I've tested it on, but it is quite ad-hoc, particularly the RMSE 20 threshold in the first portion and the entire procedure to find the rectangle coordinates.

Is there a more principled way to find these photo rectangles? How would someone with experience in image processing solve this problem?
Most of the image processing/computer vision algorithms are ad hoc. Your algorithm is good. However, I think that a few points can be improved:


Instead of finding median color, you could do use the geometry of your images, which is more robust. For example, you know that the background part (The one that touches the boundaries is black, and the one inside it has the color you are looking for.
In order to find the rectangle coordinates, you should use a blob analysis procedure. In Matlab it is done by using regionprops command. It will give you properties like 


area of blob (To remove the small ones)
solidity (Can be used to separate the rectangles from non-rectangles)
coordinates of pixels



Also, you could try to use morphological pre-processing like opening, to remove small blobs.