Why is this authentication procedure using Rabin crypto not useful?
A is acting as a square-root oracle in that protocol. We can use that oracle to factor $n$ and break the scheme. 

Suppose you are an attacker that wants to impersonate A. You:


Pick a random $m$;
Send $m^2$ to A;
Compute $p = \gcd(m_1 - m, n)$, thus factoring $n$.


This works with probability $1/2$ for each attempt.