Why is Views ignoring nid as second argument?
It seems that views ignores the second argument in case the first argument is not present, although the first argument is configured to show all values if it is not present. Setting the following PHP code as default argument for the taxonomy term solves my problem:

if (arg(0) != 'taxonomy') {
  return 'all';
} else {
  return arg(2);
}