What are the name and rules of this hand signing spell game?
Sounds a lot like Waving Hands, or Spellcaster. Here's an online copy of the rules.

According to BoardGameGeek, you can even play it online, here.