What is the mathematical property stating that it is hard to find a collision in the AES algorithm?
As mentioned in several of the other answers, AES is a family of permutations indexed by the key. Hence there cannot exist any collision of the form $AES_k(x) = AES_k(x')$ for $x\neq x'$. 

On the other hand, if we look at the function $f_x: k \rightarrow AES_k(x)$ we should ideally have a random function. Finding a non-random property of this function would be a major step in cryptanalysis of AES. So, with this reasoning the standard reasoning for random functions applies. For AES this means we have a function that maps 256 (192, or 128) bit strings to 128 bit strings. By the birthday paradox we know that an adversary that evaluates $f_x$ $q$ times has a success probability of $\mathcal{O}(\frac{q^2}{2^{128}})$ and that is also the best one can do without a huge improvement in the cryptanalysis of AES. However, this means one can find collisions of that function in about $2^{64}$ AES evaluations. This is already doable: the bitcoin network currently does about $2^{70}$ SHA2 computations per day!  And AES is faster than SHA2.