Use STM32 BOOT0 to boot custom bootloader instead of ST bootlaoder
No, the documention makes it quite clear that such behavior is fixed.

But you can use a different GPIO pin of your choice to control your bootloader's behavior - ie, boot the target firmware or stay in bootloader mode.

You could also use commands issued to the stock factory ROM bootloader to start your custom one, but that seems a little pointless as at that point you might as well just use the stock one.