Extracting motion vectors from H.264 bitstream
I think the following code can help you:
https://github.com/vadimkantorov/mpegflow
The main idea is that you need to decode frames first and query motion frames by calling av_frame_get_side_data function.