What are the flaws in this encryption algorithm?
This is not a secure encryption scheme.  It is similar to a Hill cipher, and vulnerable to similar attacks.  For instance, it is vulnerable to known-plaintext attacks: an attacker who observes a ciphertext E and knows the corresponding message M can recover the secret key and thus decrypt all other messages that were encrypted with the same key.

The recipient will be able to decrypt if A is invertible.  If A is not invertible, then the recipient won't be able to decrypt.

See also the questions about the Hill cipher on Crypto.SE.