Why does this RSA example break when: 1) setting p or q as non-prime, or 2) setting p and q as the same number? This answer talks about how non-prime numbers would make the algorithm easier to break, not that the algorithm doesn't work full-stop:
Why does RSA need p and q to be prime numbers?

Question 1:
this implementation uses the number 57, which is not prime as it divides by 3.


Choose p = 13, q = 57, e = 5 and message 'm' = 4
Therefore n = pq = 741 and (p - 1)(q - 1) = 672
encrypt message m as m^e mod n = 4^5 mod 741 = 283
calculate decryption key d as e^-1 mod (p–1)(q–1) = 5^-1 mod 812 = 269
decrypt '283' by 283^d mod n = 283^269 mod 741 = 199 ≠ 4


How does using a non-prime number cause the result to be wrong?
I suspect the answer has something to do with the way that an inverse mod calculation looks for prime factors using the Euclidean algorithm, and if n is not the product of two prime numbers then it will give a different answer. But how is (p-1)(n-1) affected by p or q not being prime?

Question 2: p and q are the same prime number:


Choose p = 11, q = 11, e = 3 and message 'm' = 2
Therefore n = pq = 121 and (p - 1)(q - 1) = 100
encrypt message m as m^e mod n = 2^2 mod 121 = 8
calculate decryption key d as e^-1 mod (p–1)(q–1) = 5^-1 mod 100 = 67
decrypt '8' by 8^d mod n = 8^67 mod 121 = 24 ≠ 4


How does using two (prime) identical numbers cause the result to be wrong? For this one I don't know where to start.
A first step is to use the correct value for
the totient function
$\phi(n),$ which is $432$ and $110$ in your examples. 

Even better use the 
Carmichael function $\lambda(n)$ as described in RSA key generation, which is
$36$ and $110.$ 

With either of these numbers, your encrypted messages are decrypted correctly.