Converting matrix to spatio-temporal feature vector
I have no experience on video data. But I had a sense that your problem is the same/or very similar to text summarization problem. 
In principle, what you want is to "summarize" all your different time frames into just only one frame. It is very much like you have many sentences in one article but you just want to have one sentence which can summarize all the sentences in one article. 
In general there are two methods:
1. Extract methods: :Evaluate the importance of each frame and put weights on different frames. Then simply use linear combination to have a summarize vector. In extreme case, you can just use the dominant frame as a representative frame if its weight is much bigger than the others. This method is very similar to TextRank or TextTeasing algorithm in NLP. 
2. Abstract methods: : In case of NLP, you need some knowledge of linguist and sentence semantic structures in order to develop this approach. In your case, I believe you need to do the same thing for your video data.
Given a matrix representing a video, where the rows represent video frames and the columns represent frame features, are there any techniques I can use to create a single feature vector from this without losing the spatio-temporal information from the matrix? i.e. 40 frame vectors with 36 features each converted to 1 vector with 36 features representing the entire 40 frames.
I've already tried taking the mean for each feature but I'm wondering about other techniques.