AVR- sending application program to bootloader
Regarding the HEX format:

As suggested in the comment, have a read about the Intel HEX format. http://en.wikipedia.org/wiki/Intel_HEX

What it says is basically, that each line corresponds to a block of data.
The format of the line being as follows:

:XYZC


Where:


X is 1 byte specifying the payload length
Y is 1 byte specifying the payload type *
Z is the payload of length X bytes
C is 1 byte of the line's checksum


*


  ’00’ Data Record 
  ’01’ End of File Record 
  ’02’ Extended Segment Address 
  ’03’ Start Segment Address Record 
  ’04’ Extended Linear Address Record 
  ’05’ Start Linear Address Record


See http://microsym.com/editor/assets/intelhex.pdf for the entire specification.



About the method of transmission:

It is also mentioned in the comment - You are much better off sending the entire file as-is. That way you don't need a specialized program at the PC-side, just a terminal that talks to your device. Then parse the text on-chip.



How to write it:

The easiest method is to start reading Atmel's application notes:


  All Self-programming operations are performed using the SPM instruction


This can be found here: http://www.atmel.com/images/doc1644.pdf

The AVR109 is a famous bootloader for Mega devices. You may find lots of info here.