Looking for a Non-textual hashing function
This is a pure encoding problem, as @kelalaka said. Take any reasonable hash algorithm like SHA-512. You have all qualities that you mentioned - not reversable, low probability of collisions, etc. What needs top be done, is to encode this hash into a picture.
To algorithm: look for instance at automatic generation of avatars at Stack Exchange to get one of possible simple ideas.
If you use SHA-512: The hash consists of 64 bytes. Consider it as a matrix of 8x8 elements, for each element define 256 values. These 256 values can be a combination of simple elements, e.g. 4 elements each having 4 states.