How to deal with changing video frame sizes in a CNN?
In your example, you are seeing the effect of a stabilisation algorithm applied to otherwise jerky movement - and the effect is a moving "window" inside the video frame. 

The actual video file has a fixed frame size. As far as I know there are no video formats that switch size frame to frame. You may have different areas of focus to worry about, but should never need to worry about frame size switching on you mid video. You can rely on your video processing library to present a consistent frame size for the whole of a video.

If you find that the border of the view inside the video frame interferes with the performance of your CNN once trained, then you could deal with it in a few ways. A couple of simple approaches might be:

1) Crop out a centre portion that always reliably has content in, and use just that portion for your input features.

2) Detect and flood-fill the black border with the mean pixel value, using an image/video library (like OpenCV) so that edges around it cause less activity in your CNN.