Can RL learn to mimic a hash function?
Has there been put any thought into it?


Probably not at a formal research level, it is basically a dead-end, as you have pointed out there will essentially be one correct solution amongst a large number of incorrect ones. No useful heuristic exists to assess partial solutions, so logically RL will not work, as an agent needs to experience (or simulate accurately) success and failure in order to learn correctly. This is a hard limit to all learning algorithms, they can only learn from available data - any combinatorial problem with one special permutation, and zero clues that disambiguate it from a large number of similar permutations is inherently unlearnable.

Only approaches that would work against a generic hash algorithm are brute-force searches that try every possible combination of operations suspected of being in the algorithm, up to a certain size of program. For any real-world secure hashing algorithm, this would be a huge search space, and intractable. For smaller insecure hashing algorithms, it is more feasible, but there would likely be no value to even demonstrating it was possible.

Related to this (when trying to reverse a hash or find collisions, not figure out the algorithm from examples), cryptographic hashes, such as MD5, SHA-1, SHA-256 etc, are designed such that brute-forcing them is the only way to solve them. In the case of MD5 and SHA-1, that has since been broken mathematically, and they are weaker then brute-forcing, but this was done starting with knowledge of the algorithm. Current RL agents and neural networks cannot match a human in learning and understanding abstract mathematics, so that approach is ruled out at least for now.