What's the usage of the three boot mode of STM32f103?
Booting from System Memory invokes the on-chip bootloader, which is present in the chip directly from the factory, before you've programmed anything into the on-chip flash.  This allows you to load (program) code into the device from an external interface such as UART or USB.  This is explained in detail in the manual.

Main flash memory is where your code typically goes.  In normal operation, your code will reside in flash, and on Power On Reset (POR), the CPU will fetch the reset vector and initial stack pointer (SP) from flash.  You can load flash via JTAG, on-chip bootloader (above), etc.

Lastly, you can load code into RAM (JTAG, runtime) and then boot/run from there.  This isn't often used, usually you're doing something tricky like a temporary bootloader or the like.