Possible ways to determine the direction of moving object? I'm trying to determine the moving direction of an object in x-axis only. Figure below shows part of my effort.

However, I got stuck. As you can see from the middle image, the green represent the dot moved to right (positive x), the black represent overlapped area and the purple represent the reference image relative to the moving image.
In bottom image, I got two directions for that one object. Left one is going through positive transition, and right negative transition. I mainly use the intensity differences to determine the direction., i.e. Moving image intensity - reference image intensity.
Is there other possible ways to determine the direction of a moving object (to and fro in x-direction only)? Or is there some ways of correction I can apply to?
Thanks in advance guys!
If your images are as clear (I specify clarity here because it is easy to threshold in this case) as your examples here, one easy way of computing the direction vector of a moving object would be to calculate its center of mass across frames. 

For this it's as simple as thresholding first (pixels which are part of the object are equal to 1, all other pixels in the background are 0), then taking the average x-coordinate of these set pixels. Find this average x-value for all your frames, and just take the difference between consecutive frames. You determine whether a positive difference corresponds to movement in the left or right direction.