Arduino New Optiboot Bootloader on custom ATMega328p board, ATMEL Studio and AVRdude
Since I am not so familiar with bootloaders, I would skip for now writing my own, so I would like to use the newest Arduino Bootloader for ATMega328p to support the firmware updates.


Even people who do know about bootloaders rarely bother writing their own when there is a perfectly good one with good support and good cross-platform tools available.  Why would you?


  The question here is, do I need to make any linker modifications or modifications in the code in Atmel Studio projects in order to be able to "transform" my code into application code so that can coexist with the bootloader?


No. The bootloader resides at a high memory address, and your code at address 0x0000 just like normal.  A fuse is set to say "Boot at the bootloader address", and another lets you set the size of the bootloader area (and thus the bootloader address).


  Since the optiboot bootloader used by Arduino uses the STK500 communication protocol and functionality, could I use AVRdude to communicate with the bootloader and flash the applications code only through UART?


Yes. This is what the Arduino IDE does. It just runs avrdude to communicate with the bootloader.  If you turn on verbose mode in the preferences you should be able to see what the exact command is that it runs so you can replicate it in your own program or script.