Track a pixel between frames using speed (vector) outputs?
I've found a solution to my own problem after some research. We need to save all four channels, because


  R: pixel displacement in X from current frame to previous frame
  
  G: pixel displacement in Y from current frame to previous frame
  
  B: pixel displacement in X from current frame to next frame
  
  A: pixel displacement in Y from current frame to next frame


as discussed here.

Updates after hours of debugging (sigh...) -- The above statements are NOT 100% accurate. The first two are accurate, whereas the bottom two should instead be


  B: pixel displacement in X from next frame to current frame
  
  A: pixel displacement in Y from next frame to current frame


Caveat -- the y-axis seems to be pointing upwards. I need to flip its sign to display correctly on my image. 

I used this MATLAB package to read all four channels of the OpenEXR images.