Using motor encoder with VESC
From your pictures, I have an opininon that this VESC can connect different type of encoders:


Hall type (HALL)
Quadrature incremental with zero index (ABI)
SPI encoder (AS5047)


Your encoder has embedded two types: quadrature AB no index, and hall. AB are quadrature incremental encoder output, while U,V,W are hall encoder outputs. From your pic, the U,V,W are going to H1, H2, H3. To not be confused with U,V,W motor power output. 

Hall encoders are absolute type each sensor is placed at 120 degrees electrical angle. Quadrature can be arbitrary resolution, so you should somewhere configure the pulses per revolution and look in the manual if it can work without index (zero) signal (IMO no).

You have to watch out the specified voltage range of the encoders.