sending data over bluetooth to my hardware through a matlab serial GUI
Nothing happens because nothing beyond the 

return U0RBR;

in the ISR gets executed.  All of your conditionals that could do something are in the dead code following the unconditional return.

Likely you need to rethink the architecture of your program, particularly the relationship between the ISR and the main program loop - there are advantages and disadvantages to doing command processing in either place.