Studies of how people learn to solve puzzles
I'm not sure if this is what you are looking for, but the big umbrella term in the literature for puzzles like the Tower of Hanoi is problem solving. 

A Google Scholar search for "tower of hanoi" and "problem solving" generates ~5000 hits. Another related puzzle is the Tower of London, and that Scholar search generates an additional ~5000.