Soft power off with delay after flipping physical switch
There are two issues with your schematic. You need a power P-channel MOSFET to switch the power to U4. You can command it with a small N-channel MOSFET (M2) like this:



simulate this circuit &ndash; Schematic created using CircuitLab
When M2 is off, M1 is also off (as R1 pulls its gate to source).
R2 keeps M2 off for the brief period when the ATmega starts up at powerup. You may omit it if you want.
The second issue (credits to Abel, I didn't see it initially) is that the Vbatt voltage is too high for a MCU GPIO pin. You need to use a voltage divider to reduce it to ~5V. This needs to be true for the whole range of the battery voltage. E.g. target 5.2V when the battery is truly 100% charged, so when it's nearly depleted it still near 5V. I've used values for R3/R4 assuming 4-cell Li-Ion/Li-Poly.
Hope this helps!
EDIT
As you indicate that battery voltage measurement is not done by the ATmega, I've redrawn the whole schematic to avoid the voltage divider, as it drains unnecessary power for no good:



simulate this circuit
The switch is wired to the MCU only. Use the weak pull-up functionality on your MCU at GPIO1 to sense the switch (or add an external pull-up to the 5V).