Can the Arduino (Nano) bootloader be updated via itself and the Arduino IDE or is an ISP required?
On ATmega328p and similar ATmega MCUs with bootloader support, the application is not allowed to write to flash. Only code running in bootloader area of the flash memory is allowed to write to flash. Bootloader area is at the end of the flash and the beginning is set with the BOOTSZ fuses.

The write to flash page is done in two steps. First step is deleting the contents of the flash page, second step is write the new data to the flash page. In bootloader area* it is allowed to first prepare the data to temporary buffer page, then delete the flash and write the data in one step. So in theory the bootloader could overwrite itself with a function running in the last flash page. But I don't know about bootloader which would do it.

To write a bootloader an ISP programmer is necessary. Or other Arduino with the "Arduino as ISP" sketch.

*note: the read-while-write area at the end of flash has the size of the largest bootloader area possible and is not determined by BOOTSZ.