Activity Monitor showing a process that takes 600% CPU?
Nothing is wrong with your Activity Monitor. 100% is the throughput on one core of a CPU.

Newer Macs (all since 2007?) have CPUs that have more than one core and or also hyper threading. Handbrake can make use of these cores and excite in parallel on more than one core.