Determining the rotation of square given a list of points
Rev 1: Ruby, 354 bytes

further golfing thanks to blutorange.

-&gt;a{t=s=Math::PI/18E4
d=r=c=0
a=a.map{|e|e-a[0]}
0.upto(36E4){|i|b=a.map{|e|(e/Complex.polar(1,i*s)).rect}.transpose
m,n=b
if n.min&gt;=f=0
l=[m.max-x=m.min,n.max].max
a.each_index{|j|f+=((l-w=n[j])*(x+l-v=m[j])*(x-v)*w)**2}
(1E-9&gt;q=f/l**8)&amp;&amp;(c&gt;0&amp;&amp;(i-d)%9E4%89E3&gt;1E3?c=9E9:0;c+=1;d=i)
q&lt;t&amp;&amp;(r=i)&amp;&amp;t=q;end}
c&lt;101&amp;&amp;a[1]?c&lt;1?'impossible':r%9E4/1.0E3:'unknown'}


Ruby, 392 bytes

-&gt;(a){
s=Math::PI/18E4
t=1
d=r=c=0
a=a.map{|e|e-a[0]}
(0..36E4).each{|i|
b=a.map{|e|(e/Complex.polar(1,i*s)).rect}.transpose
m=b[0]
n=b[1]
x=m.min
if n.min&gt;=0
l=[m.max-x,n.max].max
f=0
a.each_index{|j|f+=((l-n[j])*(x+l-m[j])*(x-m[j])*n[j])**2}
q=f/l**8
if q&lt;1E-9
c&gt;0&amp;&amp;(i-d)%9E4%89E3&gt;1E3?(c=9E9):0
c+=1
d=i
end
if q&lt;t
r=i
t=q
end
end
}
c&gt;100||a.size&lt;2?'unknown':c&lt;1? 'impossible':r%9E4/1.0E3
}


The algorithm is as follows:

-Pick an arbitrary point (the first one) and move that to the origin (subtract the coordinates of this point from all points in the list.)

-Try all rotations of the square about the origin in 0.001 degree increments, through 360 degrees.

-For a given rotation, if all points are above the y axis, draw the smallest possible square around all the points, incorporating the lowest and leftmost point.

-Check if all points are on the edge. This is done with a soft calculation that takes each point, finds the squared distances from all edges, and multiplies them together. This gives a good fit rather than a yes/no answer. It is interpreted that a solution is found if this product divided by sidelength^8 is less than 1E-9. In practice this is less than a degree of tolerance.

-The best fit is taken mod 90 degrees and reported as the correct angle. 

Currently the code returns a value of ambiguous if over 100 solutions are found (at 0.001 degree resolution. That's 0.1 degrees of tolerance.) 

first fully working function, in test program

I left the resolution at 1/10th of the required resolution to make the speed reasonable. There is an error of 0.01 degress on the very last test case.

g=-&gt;(a){
 s=Math::PI/18000
 t=1
 d=r=-1
 c=0
 a=a.map{|e| e-a[0]} 

 (0..36000).each{|i| 
    b=a.map{|e|(e/Complex.polar(1,i*s)).rect}.transpose

    m=b[0]
    n=b[1]
    x=m.min

    if n.min&gt;=0

       l=[m.max-x,n.max].max
       f=0
       a.each_index{|j|f+=((l-n[j])*(x+l-m[j])*(x-m[j])*n[j])**2}
       q=f/l**8

       if q&lt;1E-9

         j=(i-d)%9000
         c&gt;0&amp;&amp;j&gt;100&amp;&amp;j&lt;8900?(c=9E9):0 
         c+=1
         d=i
       end  

       if q&lt;t
         r=i
         t=q
       end

     end    
  }

 print "t=",t,"   r=",r,"     c=",c,"    d=",d,"\n"
 p c&gt;100||a.size&lt;2?'unknown':c&lt;1? 'impossible':r%9000/100.0   
}


#ambiguous
#g.call([Complex(0,0)])
#g.call([Complex(0,0),Complex(1,0)])
#g.call([Complex(0,0),Complex(1,0),Complex(0,1)])
#g.call([Complex(0,0),Complex(1,0),Complex(0,1),Complex(1,1)])
#g.call([Complex(0,1),Complex(0,2),Complex(1,0),Complex(1,3),Complex(2,0),Complex(2,3),Complex(3,1),Complex(3,2)])

#impossible
#g.call([Complex(0,0),Complex(1,0),Complex(2,0),Complex(3,1),Complex(4,2)])
#g.call([Complex(0,0),Complex(1,0),Complex(2,0),Complex(1,1)])
#g.call([Complex(0,1),Complex(0,2),Complex(1,0),Complex(1,3),Complex(2,0),Complex(2,3),Complex(3,1),Complex(3,2),Complex(2,2)])
#g.call([Complex(2,0),Complex(0,1),Complex(2,2),Complex(0,3)])
#g.call([Complex(0,0),Complex(2,1),Complex(0,2),Complex(2,2),Complex(-1,1)])

#possible
g.call([Complex(0,0),Complex(1,0),Complex(2,0)])
g.call([Complex(0,0),Complex(0.3,0.3),Complex(0.6,0.6)]) #(should return 45)
g.call([Complex(0,0),Complex(0.1,0.2),Complex(0.2,0.4)]) #(should return appx 63.435 (the real value is arctan(2)))
g.call([Complex(0,0),Complex(0,1),Complex(2,1),Complex(2,2)])
g.call([Complex(0,1),Complex(0,2),Complex(1,0),Complex(1,4),Complex(2,0),Complex(2,4),Complex(4,1),Complex(4,3)])


golfed version, resolution compliant with spec, takes about a minute per call, in test program.

There's still a pesky error of 0.001 degrees on the last test case. Increasing resolution further would probably eliminate it. 

g=-&gt;(a){                                                            #take an array of complex numbers as input
  s=Math::PI/18E4                                                   #step size PI/180000
  t=1                                                               #best fit found so far
  d=r=c=0                                                           #angles of (d) last valid result, (r) best fit; c= hit counter
  a=a.map{|e|e-a[0]}                                                #move shape so that first point coincides with origin
  (0..36E4).each{|i|                                                #0..360000
    b=a.map{|e|(e/Complex.polar(1,i*s)).rect}.transpose             #rotate each element by dividing by unit vector of angle i*s, convert to array... 
    m=b[0]                                                          #...transpose array [[x1,y1]..[xn,yn]] to [[x1..xn],[y1..yn]]...
    n=b[1]                                                          #...and assign to variables m and n 
    x=m.min                                                         #find leftmost point
    if n.min&gt;=0                                                     #if all points are above x axis
       l=[m.max-x,n.max].max                                        #find the sidelength of smallest square in which they will fit
       f=0                                                          #f= accumulator for errors. For each point
       a.each_index{|j|f+=((l-n[j])*(x+l-m[j])*(x-m[j])*n[j])**2}   #...add to f the product of the squared distances from each side of the smallest square containing all points
       q=f/l**8                                                     #q= f normalized with respect to the sidelength.
       if q&lt;1E-9                                                    #consider a hit if &lt;1E-9
         c&gt;0&amp;&amp;(i-d)%9E4%89E3&gt;1E3?(c=9E9):0                          #if at least one point is already found, and the difference between this hit and the last exceeds+/-1 deg (mod 90), set c to a high value
         c+=1                                                       #increment hit count by 1 (this catches infinitely varible cases)
         d=i                                                        #store the current hit in d
       end  
       if q&lt;t                                                       #if current fit is better than previous one
        r=i                                                         #store the new angle
        t=q                                                         #and revise t to the new best fit.
       end             
    end
  }
  c&gt;100||a.size&lt;2?'unknown':c&lt;1? 'impossible':r%9E4/1.0E3           #calculate and return value, taking special care of case where single point given.
}
#ambiguous
puts g.call([Complex(0,0)])
puts g.call([Complex(0,0),Complex(1,0)])
puts g.call([Complex(0,0),Complex(1,0),Complex(0,1)])
puts g.call([Complex(0,0),Complex(1,0),Complex(0,1),Complex(1,1)])
puts g.call([Complex(0,1),Complex(0,2),Complex(1,0),Complex(1,3),Complex(2,0),Complex(2,3),Complex(3,1),Complex(3,2)])

#impossible
puts g.call([Complex(0,0),Complex(1,0),Complex(2,0),Complex(3,1),Complex(4,2)])
puts g.call([Complex(0,0),Complex(1,0),Complex(2,0),Complex(1,1)])
puts g.call([Complex(0,1),Complex(0,2),Complex(1,0),Complex(1,3),Complex(2,0),Complex(2,3),Complex(3,1),Complex(3,2),Complex(2,2)])
puts g.call([Complex(2,0),Complex(0,1),Complex(2,2),Complex(0,3)])
puts g.call([Complex(0,0),Complex(2,1),Complex(0,2),Complex(2,2),Complex(-1,1)])

#possible
puts g.call([Complex(0,0),Complex(1,0),Complex(2,0)])
puts g.call([Complex(0,0),Complex(0.3,0.3),Complex(0.6,0.6)]) #(should return 45)
puts g.call([Complex(0,0),Complex(0.1,0.2),Complex(0.2,0.4)]) #(should return appx 63.435 (the real value is arctan(2)))
puts g.call([Complex(0,0),Complex(0,1),Complex(2,1),Complex(2,2)])
puts g.call([Complex(0,1),Complex(0,2),Complex(1,0),Complex(1,4),Complex(2,0),Complex(2,4),Complex(4,1),Complex(4,3)])


Note that for about 30% more code this algorithm could be adapted to work fast: it is obvious that in cases with a finite number of solutions, one of the edges lies flat along a a cube, so all we really have to try is those angles that correspond to each pair of vertices.  It would also be necessary to do a bit of wiggling to check there aren't there aren't infinitely many solutions.