What is the access record "content_access: all"? This is overriding my taxonomy access module
The access record you are seeing could be the default access that Drupal uses and that, as far as I can say looking at the code, is only used when no module implements node access rights.
node_access_rebuild() and node_access_acquire_grants() both add that access record if no module implements any node access hook, or no module returns any access record. 

function node_access_rebuild($batch_mode = FALSE) {
  db_query("DELETE FROM {node_access}");
  // Only recalculate if the site is using a node_access module.
  if (count(module_implements('node_grants'))) {
    // …
  else {
    // Not using any node_access modules. Add the default grant.
    db_query("INSERT INTO {node_access} VALUES (0, 0, 'all', 1, 0, 0)");
  }

  if (!isset($batch)) {
    drupal_set_message(t('Content permissions have been rebuilt.'));
    node_access_needs_rebuild(FALSE);
    cache_clear_all();
  }
}


I would not suggest to remove that access record from the node access table, otherwise users without specific permissions would not be able to view any node for which a module doesn't have a node access record. In Drupal, it is default to deny the access to a node; in fact, the node module writes in the node access table only the records that allow to access a node.

Update: As you reported that you are using also Content Access, you could be interested in this issue report: Do not hijack the 'all' realm.
The access record you are noticing is probably the record added by Content Access; the short name of the module is content_access, and it is probable the module is using that as grant realm.


  I think that I have to give people access to see pages by default and then take it away if taxonomy is assigned.


As Drupal node access works, the access is denied by default, and it is granted for nodes for which a module grants it; it is exactly the opposite of what you are doing.
Access is still allowed for users who have specific permissions, which in Drupal 6 means users with the administer nodes permission, and is not allowed for users without a specific permission, which in Drupal 6 means users without the access content permission.