How to keep track of the same person detected in different frames of a camera?
One way to solve the issue may be a tracker (like Kalman). It would be faster and easier approach than neural nets. 

If you insist about solving this issue through neural nets, then some magic and creativity is needed. Based on the nature of tracking, you need to feed multiple frames to predict next location of the object and check if there is any object close by. So, you may consider combining RNNs with CNNs to predict next bounding box to track and replace Kalman filter's prediction with RNNs. (check)