Standalone 328p Programming with CH340G programmer
I have confirmed that everything on the board works and that I could upload Blink through my arduino Uno via ArduinoISP but I would like to use this adapter.


Because the ATmega family does not support a partial flash erase via ISP, loading a sketch via ISP removes the bootloader. 

Therefore you must (re)load a suitable bootloader using the ISP connection, before you can perform a serial upload of a sketch using that bootloader.