How to make sure that /taxonomy/term/24/WHATEVER gives 404?
I've done something similar, but there might be some edge cases where this solution might not be suitable.

You can use hook_taxonomy_term_view() and within that check if the amount of arguments in the URL ($_GET['q']) is more than three.

/**
 * Implements hook_taxonomy_term_view().
 */
function custom_module_taxonomy_term_view($term, $view_mode, $langcode) {
  if (count(arg()) &gt; 3) {
    // This will throw a 404.
    drupal_not_found();
  }
}