Hill Cipher Plaintext Attack: Dealing with Non-uniqueness of Matrix Inverses?
More standard is that we multiply a column with a plain text vector to get the cipher text vector so the known plain text equation (the first two pairs) yield

$$KP=C = K\begin{bmatrix}5 &amp; 8\\17 &amp; 3\end{bmatrix} = \begin{bmatrix}15 &amp; 2\\16 &amp; 5\end{bmatrix}$$ so I get $P=\begin{bmatrix}5 &amp; 8\\17 &amp; 3\end{bmatrix}$ which is invertible as $\det(P) = 15 - 8 \cdot 17 = 9$ and $9$ is invertible modulo $26$, with inverse $3$ as $3 \times 9 \equiv 1 \pmod{26}$.

The inverse is easily computed (as for all $2\times 2$ matrices) via 

$$P^{-1} = 3 \cdot \begin{bmatrix}3 &amp; -8\\-17 &amp; 5\end{bmatrix} = \begin{bmatrix}9 &amp; 2\\1 &amp; 15\end{bmatrix}$$

which is easily checked to be the inverse. Note that the matrix in your text uses the transpose (as it left multiplies by $1 \times 2$ matrices instead, so all equations transpose) and the one from the online calculator is not an inverse at all (which becomes obvious if we do the full matrix multiplication; only the first row looks OK). So that online calculation is wrong, and Mathematica is confused (and you made a typo: 15 should be 5 I think). 

Anyway, knowing $P^{-1}$, $$K = CP^{-1} = \begin{bmatrix}7 &amp; 8\\19 &amp; 3\end{bmatrix}$$ (your text probably gives its transpose) and this indeed checks out as $$\begin{bmatrix}7 &amp; 8\\19 &amp; 3\end{bmatrix} \begin{bmatrix}0 \\ 24\end{bmatrix}=\begin{bmatrix} 10 \\ 20\end{bmatrix}$$

so it maps the plain ay to ciphertext KU. The decryption matrix is
then $$K^{-1}=\begin{bmatrix}23 &amp; 8\\19 &amp; 19\end{bmatrix}$$ if you want to decrypt the rest (or its transpose if working by rows from the left, as before).