Feeding 3 consecutive video frames to a CNN to track a tennis ball
The researchers who made Tracknet finally open-sourced their code and will do so with dataset as soon as they settle their copyright issues.  

Here are the links:

Arxiv paper:  Tracknet Arxiv paper

Tracknet repo

If the link is broken, let me know and I'll upload the zip and tar.gz along with Readme file to Dropbox.

Here is a link to the resulting video with tracked ball.
Fed Nadal demo

It is not perfect so probably going to combine this method with blob detection, difference frames used in other papers.  Also need to take out the lines, players and other background from being detected.  

Here is the code snippet where they concatenate three frames in predict_video.py.

#combine three imgs to  (width , height, rgb*3)
X =  np.concatenate((img, img1, img2),axis=2)