Hand detection in matlab
As far as I can tell, Matlab's Computer Vision toolbox provides vision.CascadeObjectDetector for object detection, with support only for Frontal Face (CART),Frontal Face (LBP),Upper Body,Eye Pair,Single Eye,Single Eye (CART),Profile Face,Mouth,Nose. When you say:  "the rectangle is drawn around the face instead.", that is happening because you are calling CascadeObjectDetector without specifying a specific object to detect, resulting in CascadeObjectDetector setting the to FrontalFaceCART by default. You are probably doing something like this:

detector = vision.CascadeObjectDetector;


Try doing this:

detector = vision.CascadeObjectDetector('Nose');


And you'll see how the nose is detected.

You are probably then getting excited, because it would be really easy then passing 'Hand' as a parameter instead of 'Nose', or anything. But unfortunately until now CascadeObjectDetector doesn't provide a model for 'Hand'. However, there exists a way to archieve this, the last ComputerVision toolbox provides the trainCascadeObjectDetector which you can use for detecting your own models(the hand for example). You'll see that this consists of a training process where you have to supply positive and negative images to develop your own detector, further explanations are given in the links. There is another useful tool you can use for this: a GUI tool for getting easier the classification process. 

Hope this helps