STM32F411VE Custom bootloader USB de-Init issue causing Jump to app fail
Your overall issue is in trying to branch to an application after a bootloader has heavily operated the system.  This is strongly dis-recommended on the STM32, as to get it to work you have to get every aspect of the chip back to a state the application code assumes.  Likely more obvious failures include things like having the clock PLL on and selected (as required for USB) and then the application code expected the PLL is not selected when it starts configuring it.  But chasing down everything will be extremely frustrating, and you can be left with 90% functionality but 10% seemingly inexplicable weirdness in some peripheral.

Instead, the usual recommendation is to set a flag in memory or the RCC backup registers, reboot the system, and then detect the flag and branch early in startup before doing any configuration.  When I needed to implement this I did it in assembly before even the C startup had run; that was probably unnecessary though if you chose to use a memory flag beware of the likelihood of startup code initializing RAM.  If entering the bootloader requires explicit action, you wouldn't even need the flag, but only to check that the "stay in bootloader" GPIO or whatever condition is not met, and that there is an application present matching whatever validity check you use.


  boot fails when setting the MSP of the application


This is more easily explained.  Your code attempts to utilize a local stack variable that was allocated before it changed the stack pointer, which is essentially a recipe for disaster.  If you are going to do that, you don't want to make any use of the stack after you change it.