Is changing $H(R\mathbin{\Vert}A\mathbin{\Vert}M)$ to $H(R\mathbin{\Vert}A\mathbin{\Vert}H(M))$ in EdDSA secure? EdDSA calculates $s=(r+H(R\mathbin{\Vert}A\mathbin{\Vert}M)\cdot a)\bmod\ell$ and $r=H(h\mathbin{\Vert}M)$, with


$H$ being the hash function (SHA2/512);
$B$ being the generator;
$A$ being the public key;
$a$ being the secret exponent;
$r$ being the per-signature random;
$h$ being the secret seed ($h[b~.\!.~2b-1]$ in the paper).


What is the effect on the security of the signature if $M$ is replaced by $H(M)$ or $H(A\mathbin{\Vert}M)$, thus producing $s=(r+H(R\mathbin{\Vert}A\mathbin{\Vert}H(A\mathbin{\Vert}M))\cdot a)\bmod\ell$ and $r=H(h\mathbin{\Vert}H(A\mathbin{\Vert}M))$?

This would mean the message needs to be hashed only once during signing, which is more efficient for long messages, but would the signature lose any security quality? Specifically the paper says that the signature provides collision resistance, would this disappear with $M$ being replaced by $H(M)$? What about $H(A\mathbin{\Vert}M)$?
Assuming this is the paper you're talking about, your modification completely eliminates resilience to collisions in the underlying hash function $H$. The EdDSA scheme (and in the Schnorr scheme on which it is based) is highly resilient against collisions in $H$. Specifically, in the generic group model, the Schnorr scheme has been proven to be secure even if $H$ isn't collision-resistant; all it requires for security is that $H$ be resistant to random-prefix preimage attacks and random-prefix second preimage attacks (essentially, given random $R$: given $h$ it's hard to find $M$ s.t. $H(R||M)=h$, and given $N$ it's hard to find $N^\prime$ s.t. $H(R||N)=H(R||N^\prime)$ and $N\ne N^\prime$).

So EdDSA can be perfectly OK with a not collision-resistant hash; as far as we know, it doesn't need collision resistance, just random-prefix preimage and random-prefix second-preimage resistance.

However, your modification is trivially vulnerable to a collision in $H$: If $H(M)=H(M^\prime)$, then if you hash $M$ before signing, you end up with an EdDSA signature of $H(M)=H(M^\prime)$ which (under your modification) is a signature for $M^\prime$. If you instead do $H(A||M)$, an attacker who finds any chosen-prefix collision can create two messages such that a signature for one is a signature for the other. If you use $H(A||M)$ as the thing to be signed, someone who can find any two messages s.t. $H(A||M)=H(A||N)$ again can take a signature for one and use it as a valid signature for the other. With a good hash function, they can't find said collision, but Schnorr (and so EdDSA) is designed such that it doesn't break if they can do that.

If someone gets a message $M$ signed with $(R,S)$, and finds a second preimage for $H(R||A||M)$ (i.e. $M^\prime$ s.t. $H(R||A||M^\prime)=H(R||A||M)$, then the signature for $M$ is valid for $M^\prime$. However, an attacker with just the ability to find collisions cannot exploit this: they do not control $R$, and $R$ is different for each $M$. So finding a collision in $f(M)=H(R||A||M)$ is useless for the attacker; to exploit the collision, one of the colliding messages must have $H(h||M)B=R$, which is not going to be the case for a generic one of a colliding pair. Collision attacks do not apply, because a colliding pair only helps if they collide when prefixed by a secret function of one of them, which will not generally be the case.