One Time Pad OTP : Optimal modular base per given plaintext space?
I am assuming that your scheme encrypts $x$ to $x + k \bmod N$, where $k$ is the key and $N$ is the modulus.

If your modulus is less than the alphabet size, decryption does not work.

Suppose your alphabet size is $n$ and you work modulo $n-1$; then the plaintexts $0$ and $n-1$ are considered the same, since $0 \equiv n-1 \pmod{n-1}$. Thus, no matter your key, they will be encrypted in the same way, and when you try to decrypt, you cannot tell them apart.

If your modulus is greater than the alphabet size, you do not have perfect secrecy (hence not an OTP).

This was already covered in answers to the referenced question.