How to approach detection of anomalies in a video feed? As part of a project I have a video from a quadrotor that captures the road and the environment around it and I want to identify and detect potholes and speedbumps on the road using image processing. What is the best approach for this?

Here is what I've done until now.

The video feed is converted into grey scale and I use one particular portion of it(marked by the blue rectangle) for processing. In that window, I again get smaller windows of and I calculated their corresponding Grey Level co-occurrence Matrix (GLCM henceforth) of all those windows. I calculated the textural features from the GLCMs like ASM, erratic homogeneity etc and I plotted them as well. 

I did this with the hope that there will be some kind of obvious change in the features when the blue rectangle reaches the speed bump(which is slightly ahead in the image). But the results erratic and I cannot make a definite decision.

If there is any other approach, or if there's any change I could implement to get better results in the same method do let me know.
Here is some brainstorming: 

You have two interesting variables here, I would say: 


spatial changes in y-direction (left to right) 
temporal changes (t-direction).


I would not directly include the x-direction (up-down direction) in the first try, since this is correlated to the t-direction.

Try the following:


Calculate the y-gradient
Calculate the t-gradient
Create a binary mask from both using two threshold values (you probably have to figure them out in a test run), so that the whole image is zero where there is no change and one where there is a change. 
To suppress spurious noise in the binary mask, you could median-filter it with a 5-by-5 or 7-by-7 median filter (try out the best size)
Use a morphologic dilate filter to close "holes" in the binary mask and also to dilate the masks so that they do have a broader overlap


This way you are directly using the visual features of the image - changes in the gray values. Now a few things can happen with those two masks


Areas of the t and y masks overlap: You have a change in the image in y direction and there is also a change between to consecutive images - you found a localized change that could be a hole or a speedbump
There is a change in y direction, but no or just a small area in t direction: That could be a road marking (the algorithm might fail if your drone tumbles quite strongly and your camera has a low number of frames per second - then the t-gradient might be quite high in this case. To suppress those signal changes from lane markings, you could suppress lines in the binary mask by a Hough-transformation of the mask and kicking out pairs of dots)
There is change in t but not in y - probably a lane mark that started/stopped


If this really works with your data, I do not know. However, starting with simple things can lead you into a quite robust and reasonably fast direction for your image processing task.