1. A method for classifying a video file according to one or more scene classes, the video file including a plurality of frames, where each frame of the plurality of frames includes a plurality of pixels, and where each pixel of the plurality of pixels is associated with a vector of material classification scores describing material content in its respective frame, comprising: for each frame of the plurality of frames, generating one or more scene classification scores associated with each of the one or more scene classes by: dividing the frame into a plurality of grid cells; retrieving the vector of material classification scores for each pixel in the frame; for each grid cell of the plurality of grid cells, averaging the material classification scores across each pixel in the grid cell to form a material occurrence vector for the grid cell; concatenating the material occurrence vectors for each grid cell of the plurality of grid cells to generate a material arrangement vector for the frame; and based on the material arrangement vector generated for the frame, generating the one or more scene classification scores associated with each of the one or more scene classes using one or more scene classifiers; based on the one or more scene classification scores generated for each frame of the plurality of frames, generating a representative scene classification score for each of the one or more scene classes; and for each of the generated representative scene classification scores that is above a predetermined threshold value, labeling the video file according to the respective scene classes associated with the scene classification scores that are above the predetermined threshold value.