Understanding how Oscar can decrypt a message in RSA
Let's assume textbook RSA, which is not apparent from the question. In that case $b$ is the public exponent (usually this is $e$). So yes, encryption is then the first-to-last formula if $x$ is the index in the alphabet of the character. However, the $\bmod 26$ in the last formula doesn't need to be used; I think that the text is just used to indicate that $x$ is within the range $[0, 26)$. The word residue is out of place here; no division is performed on the alphabet, so there is no residue. The modulus for RSA encryption is just $n$.

There are many ways how this crypto system is vulnerable:


textbook RSA is deterministic, so you can do frequency analysis of the ciphertext if the characters are encrypted separately;
indices 0 and 1 (A and B) produce the values 0 and 1 after encryption;


but the main reason that this is insecure is that an attacker could just take the public exponent and then encrypt each character of the alphabet, creating a table from a specific ciphertext to a letter. After that you can iterate over find each ciphertext and replace it with the letter.

These attacks are not feasible if a good padding scheme such as RSA / OAEP is used.