How can I set program associations in mac?
With on-board GUI tools this isn't possible. But you can download and install RCDefaultApp which is a PrefPane.

Open System Preferences > Default Apps. Choose the Apps tab, scroll down to VLC in the left panel and choose it. In the right panel scroll down to Extensions and hit the (cycle) button...



This works for other apps too - VLC is just an example also used in the question!