Best Guess Rectangle
I must admit I haven't worked on anything like this before, but looking at your data it seems to me that computing the median of e.g. the lower left corner point of all rectangles (i.e. taking x and y dimensions into account) should get rid of outliers. After having removed outliers from your data, you could simply try to average the remaining rectangles. Obviously, such an approach requires some heuristic tweaking, but it may be worth a try.



I just tried Matt's approach (just taking the 1D medians of the top/left/bottom/right X and Y values) and the results seem pretty good. See plot below of 20 realizations of various random rectangles.



scilab CODE ONLY BELOW

function r = getRect(t,l,b,r)
    large =  [1 1 0 0]*rand(1,1,'normal')*abs(t-b) + [0 0 1 1]*rand(1,1,'normal')*abs(l-r);
    small =  [1 1 0 0]*rand(1,1,'normal')*5 + [0 0 1 1]*rand(1,1,'normal')*5;

    noise = small;
    if (rand(1,1,'uniform') &lt; 0.2) then
        noise = large;
    end
    r = [t l b r] + noise;
endfunction


for p = 1:20;

N = 20;

x_points = [2 2 4 4 2];
y_points = [1 3 3 1 1];

subplot(5,4,p)
RR= [];
for n=1:N
    R = getRect(10,10,100,50);
    RR = [RR; R];
    plot(R(x_points),R(y_points),'r');
end

RRm = median(RR,'r');

xset("thickness",4)
plot(RRm(x_points),RRm(y_points),'g');
xset("thickness",1)
end;