How to restart or shutdown a Mac when it says app running but cannot force quit the app?
Good to hear that the shutdown worked.  Remember to precede it immediately with a 'sync', executed as the root user or via the sudo analogue, to flush the hard drive caches to disk from volatile memory.  

If you desire to investigate deeper, in hopes of learning what is going on ( I hope you seek more info... ), then please do the following:

 1.  Open Terminal, and then open two, distinct windows.  


 2.  In one window only, type:  

     sudo dtruss -afloe -n VLC &gt; ~/Desktop/tracevlc.txt 2&gt;&amp;1

 3.  In the other window, type:

     open /Applications/VLC.app

 4.  Attempt to reproduce the original problem.  If you need to shutdown, do so.  

 5.  When the machine reboots, read the tracevlc.txt on your Desktop -- it might well provide a clearer understanding of what is happening...  


Hope that this helps!  Let me know if you find anything of interest...

F.

=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=
my previous response follows:
=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=

Do you get anything from executing as your login-user (non-root) the following in a Terminal:

launchctl list | grep -i vlc


or

sudo pgrep -f -l -i vlc


Try this:


obtain a root shell via:

sudo -s

Sync drive buffers to hardware:

sync

Attempt to terminate vlc by various means:

pkill -KILL -f -i -l vlc

killall -9 -z vlc 

killall -9 -z VLC.app

Attempt to tear down user-space, terminate all userspace apps, shutdown system services, and reboot.  If you are running a recent version of OS X, then this works best:

launchctl reboot system



Otherwise, do this:

    shutdown -r now