Modifying a Form submit button behavior
In a custom module, implement hook_form_FORM_ID_alter(). In there, the $form['#submit'] element is an array of all the functions that will handle a valid form submission. You can add your function to that array to have it executed on (valid) submission. Submit handler functions are executed in the order of the array, so the best is probably to append yours at the end. If the array is empty, you need to create the array and it must contains default submit handler in addition to yours, otherwise your will loose the default behavior.

To open a popup, you will need some custom JavaScript to the page trough drupal_add_js() (which you would call from your submit handler). In your custom JavaScript, you can use jQuery UI Dialog library bundled with Drupal 7. To have the library available on the page use drupal_add_library('system', 'ui.dialog'). You can use drupal_add_js() to pass data from your validation handler to your JavaScript code using the setting type.