How can I validate a fieldset or any other kind of group in drupal Form API?
You could do it with custom code by implementing hook_form_alter() and then adding a custom validation function and checking for the relevant field values (I don't know anything about your form so I'm not going to write any code).

Or, you can use the field validation module, which allows you to configure via the drupal UI, extra field validation requirements.

It is a very flexible module and a couple of its validation options are:


Require some of several fields (in the field validation extras sub-module).
Require at least one of several fields


There are many more options so it would be worth playing with it to see if you can achieve what you want.