Validate Blokus Board
Python 2, 446 422 385 384 bytes

-1 byte thanks to @ovs



l=len
A=abs
s="""def v(O,N):
 def c():
 qp:
    qp:
        if(i!=j)*any(A(t-T)==1qiqj):i+=j;p.remove(j);c();return
 p=[[a]qO];c();r=C=p[:1];i=[sum(A(b-a)*A(c-b)qPqPqP)qp];N-=1
qC:
  if l(o)>5:r=0
 qp:C+=[P]*(1-all(A(a-b)-2**0.5qoqP)-(P in C))
 print(l(p)-l(C)+l(i)-l(set(i))==0)*any(k in Oq[0,N*1j,N+N*1j,N])*r,p>[]"""
exec eval("s"+".replace('q',' for %s in ',1)"*14%tuple("ijtTaabcPoPabk"))


The output format is a bit strange: it prints more than 7 characters for truthy output, and 7 or fewer characters for falsey output (7 characters is the length of "[] True". This may be stretching it, so the p&gt;[] can be replaced by if p!=[] else 1 for +11 characters to give more traditional truthy/falsey output.

The input is a list of coordinates given as complex numbers, along with the dimension n.

Try it online!

All testcases

The invariant may be the most interesting part. Since it must be conserved on rotations and reflections, distance between corresponding tiles is just about the only thing that stays constant. Also, I could not rely on the order that the tiles are listed in the piece. Using the product/sum of distances between every tile led to collisions, but using triples of tiles worked well.

I used for _ in fourteen times (wow), so the exec/eval saves 37 bytes.

Ungolfed:

def invariant(piece):
    inv=1
    for a in piece:
        for b in piece:
            for c in piece:
                inv+=abs((b-a)*(c-b))
    return inv

def valid(occupied,n):
    # convert list of rows into coordinates of Xs
    if not len(occupied):
        return True
    # generate list of pieces (this takes care of rule #2)
    pieces = [[a] for a in occupied]
    def consolidate():
        for i in pieces:
            for j in pieces:
                # merge two pieces if they share an edge
                if i!=j and any(abs(tile1-tile2)==1 for tile1 in i for tile2 in j):
                    i += j
                    pieces.remove(j)
                    # this repeats consolidate until no change (nothing left to combine)
                    consolidate()
                    return
    consolidate()
    # get a connected component of pieces (for rule #3)
    connected = [pieces[0]]
    for connected_piece in connected:
        # eventually every piece should be connected if the board is otherwise valid, so
        # we can check for valid-size pieces in this outer loop
        # all polyominoes with at most 5 tiles are valid
        if len(connected_piece) &gt; 5:
            return False
        for piece in pieces:
            if any(abs(a-b)==abs(1+1j) for a in connected_piece for b in piece) and piece not in connected:
                connected += [piece]
    # check that all pieces are only used once
    ids = [invariant(piece) for piece in pieces]
    used_once = len(ids) == len(set(ids))
    nm = n-1
    rule_1_met = any(k in occupied for k in [0, nm*1j, nm+nm*1j, nm])
    return used_once and rule_1_met and len(pieces) == len(connected)