Add event listener to modal close
The clue was seeing the Drupal's modal just uses jquery widgets. So a simple Google for &quot;jquery widget close event&quot; lead me to this SO answer.
So the solution is:
$('#drupal-modal').on('dialogclose', function(event) {
    alert('closed');
});