How do I properly trigger the ATMega328 reset pin via software so that the bootloader runs and the firmware can be updated via RX/TX pins?
You wrote:


  void(* resetFunc) (void) = 0;


This seems to me like an approach worth trying, except that the function
address is wrong. At address zero of the flash you have the start of the
user's code. The bootloader, if you do use one, lives at the end of the
flash. The start address of the bootloader depends on its size, and is
configured by setting some fuses on the chip. If you are using the
popular optiboot (the bootloader of the Uno), you have to jump to
exactly 512&nbsp;bytes before the end of the flash, i.e.:

void (*resetFunc)(void) = 0x7e00;


Oh, wait... the flash is organized as 16-bit words, so you may have to
use the wordwise address instead:

void (*resetFunc)(void) = 0x3f00;


But do note that, unlike a real hardware reset, this approach does not
reset the peripherals to their normal initial state. You may want to do
it before jumping to the bootloader.



Edit: I have disassembled a test program that does this kind of
call-by-pointer, then double checked the documentation of the icall
instruction. It appears that the function address you provide is
indeed interpreted as a word-wise address, so you should write half the
byte-wise address. If you write anything larger than the size of the
flash (0x4000 = 16&nbsp;kilowords), the most significant bits get
lost, and the address gets reduced to the correct range modulo the flash
size.

That is, if 0x7800 worked for you, then presumably the correct address
is 0x3800, i.e. 2&nbsp;kilowords (4&nbsp;KB) before the end of the
flash. This is quite a large bootloader, and the largest size supported
by the ATmega328P. Are you sure it is righ? Aren't you sliding through
some uninitialized flash before hitting the bootloader? Here is a table
that gives the bootloader's start address as a function of the amount of
memory reserved for it:

reserved size  start address
    512 B         0x3f00
     1 KB         0x3e00
     2 KB         0x3c00
     4 KB         0x3800


Note that the Arduino Uno uses optiboot, which fits into the
smallest size that can be allocated to a bootloader (512&nbsp;bytes).