Assigning hotkeys (keyboard shortcuts) in Mac OS X 10.5.8 for audio playback
The function keys work because they are system-wide "media player control keys".  Keystrokes always go to the system, which usually passes them exclusively to the active application, but some have special handling.   There's a chance that you could reconfigure VLC so that the system-wide media key that means "rewind" does skip back instead.  Check that out.  

Otherwise you're going to need some kind of additional keyboard shortcut app like QuicKeys.  (See also these suggestions).  Don't give up yet!  You almost certainly can cobble together a solution.