How can I recover an unresponding application without quitting?
Regardless of how important the process is, if it’s hung it’s hung. Quitting the finder is one way to see if you can recover it.  Quitting all open applications is anther suggestion.

I would recommend opening up a terminal window and typing the following to see if you get any clues:

sudo lsof | grep matlab


lsof is a Unix command that basically means “list open files”. The | next to that is a pipe command which basically means output should be processed by the command to the immediate right. And grep basically filters out the output based on criteria. In this case, matlab itself.

If you run that it should list all open files which also means connected processes. If you are not used to the terminal output this could be overwhelming, but looking through the output line by line might reveal something.

And if you are up for it, I would recommend installing htop in OS X. It’s a very nice and—in my opinion—more useful version of top which shows all system basics in a live updated screen. Installing in not easy and might require Xcode, but once up and running it will help you further debug things.

For example, if you can start htop and then hit the T key to show all processes in threaded mode.  Meaning you can then see all the child processes connected to matlab and get more clues there.