add captch to registration modal box
Try something like the following:

$form['my_captcha_element'] = array(
  '#type' =&gt; 'captcha',
  '#captcha_type' =&gt; 'captcha/Math',
);


More info here :https://www.drupal.org/node/743056