Why can't the atmega328p accept 5 and 3.3v signals at the same time?
This is to expand on Olin's comment about protection diodes.
The protection diode will clamp the 5V input, and it can be damaged in the process.



(fig. 10-1 from p.60 in the datasheet for ATmega32u4.  Purple scribbles mine.)

On a different note.  From the O.P.:


  The same chip can be run at 5v or 3.3v, so it's tolerant to 5v [...]
  [emphasis mine, N.A.]


There is a flaw in this reasoning.   5V tolerant means that an IC can use 5V input while it's powered from +3.3V (or some other voltage lower than +5V).  If the IC can use a 5V input while it's powered from a +5V supply, that doesn't constitute 5V tolerance.