Making Terminal recognize HandBrakeCLI commands
Have you downloaded the command line version?

From HandBrake's CLIGuide:


  tell the shell to look for the program in the current working
  directory. To run a command in the Terminal, you just type its name.
  But the shell will only look in the places where applications are
  usually stored (like /usr/bin/). You need to tell it to look in the
  current directory. To do that, precede the program name with ./ like
  this:
  
  ./HandBrakeCLI
  
  Of course, running that won't do much anything useful. If you run
  HandBrake without telling it what to do, it will just tell you to read
  the help...


Or: Instructions on how to compile Handbrake in OS X, probably what you're looking for.