How to know whether the object is moving after it is being detected?
Either (a) re-recognize it in subsequent frames and then train a network employing a trajectory model to the changing object model parameters or (b) recognize the object and its motion in a single object-trajectory parameterized model from the sequence of sets of detected edges indicating edge movement in the sequence of frames.