Is changing $H(R\mathbin{\Vert}A\mathbin{\Vert}M)$ to $H(R\mathbin{\Vert}A\mathbin{\Vert}H(M))$ in EdDSA secure?
Assuming this is the paper you're talking about, your modification completely eliminates resilience to collisions in the underlying hash function $H$. The EdDSA scheme (and in the Schnorr scheme on which it is based) is highly resilient against collisions in $H$. Specifically, in the generic group model, the Schnorr scheme has been proven to be secure even if $H$ isn't collision-resistant; all it requires for security is that $H$ be resistant to random-prefix preimage attacks and random-prefix second preimage attacks (essentially, given random $R$: given $h$ it's hard to find $M$ s.t. $H(R||M)=h$, and given $N$ it's hard to find $N^\prime$ s.t. $H(R||N)=H(R||N^\prime)$ and $N\ne N^\prime$).

So EdDSA can be perfectly OK with a not collision-resistant hash; as far as we know, it doesn't need collision resistance, just random-prefix preimage and random-prefix second-preimage resistance.

However, your modification is trivially vulnerable to a collision in $H$: If $H(M)=H(M^\prime)$, then if you hash $M$ before signing, you end up with an EdDSA signature of $H(M)=H(M^\prime)$ which (under your modification) is a signature for $M^\prime$. If you instead do $H(A||M)$, an attacker who finds any chosen-prefix collision can create two messages such that a signature for one is a signature for the other. If you use $H(A||M)$ as the thing to be signed, someone who can find any two messages s.t. $H(A||M)=H(A||N)$ again can take a signature for one and use it as a valid signature for the other. With a good hash function, they can't find said collision, but Schnorr (and so EdDSA) is designed such that it doesn't break if they can do that.

If someone gets a message $M$ signed with $(R,S)$, and finds a second preimage for $H(R||A||M)$ (i.e. $M^\prime$ s.t. $H(R||A||M^\prime)=H(R||A||M)$, then the signature for $M$ is valid for $M^\prime$. However, an attacker with just the ability to find collisions cannot exploit this: they do not control $R$, and $R$ is different for each $M$. So finding a collision in $f(M)=H(R||A||M)$ is useless for the attacker; to exploit the collision, one of the colliding messages must have $H(h||M)B=R$, which is not going to be the case for a generic one of a colliding pair. Collision attacks do not apply, because a colliding pair only helps if they collide when prefixed by a secret function of one of them, which will not generally be the case.