How to use `adb shell screenrecord --output-format=h264` together with VLC
Your command adb shell screenrecord --output-format=h264
is missing a - at the end.
It should be

adb shell screenrecord --output-format=h264 - | /Applications/vlc.app/Contents/MacOS/VLC -