How do I prove no algorithm exists for a given problem?
The term or tag you are looking for is "undecidability" or "computability". The basic version of the theory of undecidability is a part of the theory of computation, which is an ingredient of the curriculum of all students majoring in Compute Science, as far as I know.

You can search for "undecidability" or "halting problem", the most famous and the most canonical problem that cannot be solved by an algorithm, in your textbook or course material. Or right here. There are also much stuff around.