Disable/pause audio when active playback device changes
You can achieve it with Macrodroid by creating macros to pause apps such as VLC.


Open the app, select Add Macro.
Under the Connectivity section, select Headset Insert/Remove, then choose Headphones Removed. Click OK.
Go to Actions tab, under the Media section, select Control Media, select Simulate Media Button, click OK, then select Pause. Click OK and select the app you want to mute/pause. Click check the check mark button on the bottom right, give the macro a name.


When completed, the macro will look like the image below (there are 2 Media Pause because 2 apps where selected).



A solution to disable sound only for VLC if you are unable to use the steps described above: 


Open VLC, go to Settings, then to Extra settings, click on Audio.
Go to Headset section, enable Detect headset.


Note: I didn't test it with a Bluetooth headset.