Remove an unobstructed rectangle
CJam, 241 bytes
(with newlines removed.)
rri:Hri:Vri:Q[q~]3/_Qa3*a+_|$W%:Pf{\a#}:AH/:B0ff*
P,,[AHAW%HBz:+_W%V\V]2/
ff{~@@f=/::|1#}0Ua4*t:R;
P0f&lt;
V{H{BI=J=_2$=
0R{&quot;I&gt;! I+V&lt;J&gt;! J+H&lt;&quot;4/+4/z{~~}%:&amp;1$*\)}%);2$-|t
}fJ}fI
[P,{_La#\1$0t1$f-}*;;]
{:TR=2/~\~V\-,&gt;\f{\_3$=@~H\-,&gt;{Tt}/t}~}/
:~Pf=:~
~]S*N

It uses the ppm file format. Example usage (using ImageMagick):
convert IVYvE.png -compress none ppm:-| (time /path/to/cjam-0.6.4.jar 1.cjam) |display

Well, it is too long and too slow... Runs about a minute for the example.
I resized the test cases (and added some others) to make testing easier.

  
  
  
  

It seems the color space informations are lost so the colors are slightly different.