How can I do video classification while taking into account the temporal dependencies of the frames?
Look at spatio-temporal CNNs which extend the image-based CNN in 2D to 3D to handle time.  These are commonly used to detect or classify action in a video.  People have used them to identify specific actions in various sports such as kicking a soccer ball, throwing a baseball or dribbling a basketball.  They have been used to identify fire, smoke, deep fakes, and violence in videos.  Below are some articles to help get started.

Introduction to Video Classification, by Connor Shorten
Are you ready for a Video Classification Challenge? Preparation Guide for Video Classification, by Neha Goel
MiCT-Net for Human Action Recognition in Videos, by Florent Mahoudeau

Source code:

Video Classification
Video Classification Using 3D ResNet