Hashing a video file
Extracting one single hash per video and expecting it to work for subsegment matching is unrealistic.

The approach - which is the one used in audio identification systems but could be extended to video - is to extract many hashes, one for each small segment of content (say a few seconds of video). The matching process consists in matching your "query" sequence of hashes to the sequence of hashes extracted from your video. This is quite similar to a text indexing/retrieval problem, and traditional data structure for such problems (like inverted indices) would work. You can probably build your implementation on top of an existing text indexing system like Lucene/Solr. See for example, the echoprint server code, built on Solr.

This addresses the sequence aspect. Remains the problem of generating robust hashes for images. What you are doing is probably going to still be sensitive to minor differences in compression/resolution - You'll need a very rough quantization of a small number of DCT / wavelet coefficients for it to be robust.