How to approach detection of anomalies in a video feed?
Here is some brainstorming: 

You have two interesting variables here, I would say: 


spatial changes in y-direction (left to right) 
temporal changes (t-direction).


I would not directly include the x-direction (up-down direction) in the first try, since this is correlated to the t-direction.

Try the following:


Calculate the y-gradient
Calculate the t-gradient
Create a binary mask from both using two threshold values (you probably have to figure them out in a test run), so that the whole image is zero where there is no change and one where there is a change. 
To suppress spurious noise in the binary mask, you could median-filter it with a 5-by-5 or 7-by-7 median filter (try out the best size)
Use a morphologic dilate filter to close "holes" in the binary mask and also to dilate the masks so that they do have a broader overlap


This way you are directly using the visual features of the image - changes in the gray values. Now a few things can happen with those two masks


Areas of the t and y masks overlap: You have a change in the image in y direction and there is also a change between to consecutive images - you found a localized change that could be a hole or a speedbump
There is a change in y direction, but no or just a small area in t direction: That could be a road marking (the algorithm might fail if your drone tumbles quite strongly and your camera has a low number of frames per second - then the t-gradient might be quite high in this case. To suppress those signal changes from lane markings, you could suppress lines in the binary mask by a Hough-transformation of the mask and kicking out pairs of dots)
There is change in t but not in y - probably a lane mark that started/stopped


If this really works with your data, I do not know. However, starting with simple things can lead you into a quite robust and reasonably fast direction for your image processing task.