Boot loader for updating Firmware
In smart phones and other similar devices, the bootloader will not do any kind of communications to download the new software image: this will be handled by a 'download client' that runs in the application and stores the image to a memory location that can be used by the bootloader.  Once the image has been downloaded and validated, the application resets the device to launch the bootloader.

This is a good approach for several reasons:


The bootloader is frequently constrained in size and coding it as a
simple Flash programmer makes this less restrictive.
The application has the opportunity to inspect the downloaded image for errors (CRC,
Hamming codes, etc) before programming starts.
Interrupting the download is less likely to result in a device failure.
You can use multiple physical transport media for performing the download.
You can unit test the bootloader by provisioning test images without
having to test the transport at the same time.
The bootlaoder can verify the application software every time the device is reset or powered and re-program it if required.


The main drawbacks are:


You may need an external non-volatile memory device to store the downloaded image.
The bootloader will need to be able to access the external memory device to retieve the dowloaded image.


I would therefore recommend that you move the USB and Serial download code from the bootloader to the application, implement an external memory device to store the downloaded software and convert the bootloader to be a 'NVM bootloader'.