How to know whether the object is moving after it is being detected? If my algorithm detects the type of object, how should I know if that object is moving or not? Suppose a person carrying an umbrella. How to know that the umbrella is moving?

I am working on a project where I want to know whether that particular object belongs to the person entering inside the store.
I was thinking about the bounding boxes(bb) approach where if the person's bb overlaps with the object's bb. But the problem arises when there are multiple objects with a person. 

I would appreciate your help. Thanks.
Either (a) re-recognize it in subsequent frames and then train a network employing a trajectory model to the changing object model parameters or (b) recognize the object and its motion in a single object-trajectory parameterized model from the sequence of sets of detected edges indicating edge movement in the sequence of frames.