Is there any way to create validation function in system_settings_form
The system_settings_form should be used to set simple variables for your module, see the approved answer and related comment here:

Is &quot;system_settings_form&quot; voodoo?

If you want to be sure both form fields are filled out during submission you don't need a validate function -- you can set both fields as being required when you create your form array:

$form['SOMETHING']['#required'] = TRUE;


where something is your form field key.

If you have complex logic and your form isn't simply setting a configuration variable you will call with variable_get or variable_set later -- then implement a custom form_validate and form_submit function for your form as your form is not simply a variable settings form.

See the Form API Handbook. Also of use is this drupal $form lifecycle or workflow illustration. Lastly form validation helpers and submit handlers are touched on briefly on this page of the handbook.

EDIT:

Just to complete my answer, hook_form_alter can change many aspects of a form used by drupal.

EDIT 2:

You can add a custom validator on the form you use to build return system_settings_form($form); for an example see the Commerce Admin include file here: http://cgit.drupalcode.org/commerce/tree/includes/commerce_ui.admin.inc

They set:

$form['#validate'][] = 'commerce_currency_settings_form_validate';


in their form builder function then they provide a validation function:

/**
 * Form validate handler for the currency settings form.
 */
function commerce_currency_settings_form_validate($form, &amp;$form_state) {
  // Ensure the default currency is always enabled.
  $default = $form_state['values']['commerce_default_currency'];
  $form_state['values']['commerce_enabled_currencies'][$default] = $default;
}