No error messages from validate?
For fieldsets I think you have to include the parent as well.

Try changing 

form_set_error('phone', 'Ditt nummer v.... 
to 

form_set_error('contact][phone', 'Ditt nummer v....

You can read more here: http://drupal.org/node/678816