Is there a specific term for the notion of storing "algorithms" in human memory?
This is called procedural memory.

In textbooks, memory is often broken down into a hierarchy of types.  Note that this taxonomy is primarily a guide to language use - ie, how types of memory are labelled or referred to, not how memory is actually organised in the brain.  In standard hierarchies, recalling a mathematical formula would fall under semantic memory, while the process of applying the formula would be stored in procedural memory.



Procedural memory is considered implicit (unconscious) memory, and is most often associated with motor skill or muscle memory, but includes any learned procedure or process.  Notably, the process of learning a motor skill (such as riding a bike) mirrors that of learning a cognitive skill (such as playing chess), in that initially, the process can be acquired consciously (explicit), but with practice, it becomes unconscious (implicit).