What is the best way to compare a subjects video file to a csv file of data we have?
I think it is best if you consult a professional data scientist. However, I can think of one way to approach your problem.
Because a video is what data scientists call unstructured data, you'll have a hard time analyzing it raw.
What I would do is to define some key points during the gameplay (ie. beginning of a new level or confronting a new puzzle), create a time interval around that key point, and chart the button pushed based on these intervals.
Consider a set of gameplay videos with 2 key points. For example, the key point in one of the videos is set in times 04:13 and 32:00. So the video's timeline should look something like this.

(Begining) 00:00 --------> 04:13 --------> 32:00 --------> 45:00 (End)

Now for each key point, define a time interval of say, 2 minutes and record if the button is pushed in the obtained time intervals. So for key point number one, you will have 02:13 until 06:13. After doing so for every video and adding up all the data obtained, You'll end up with a table like below:




Time Intervals
Time interval for 1 player
#Buttons Pushed




Interval No.1
00:00 - 02:13
5


Key Point No.1
02:13 - 06:13
19


Interval No.3
06:13 - 30:00
7


Key Point No.2
30:00 - 34:00
25


Interval No.4
34:00 - 45:00
9




This basically means that for example, 7 people pushed the button during the time interval no.3 which in the above example (for that one player) is around times 06:13 - 30:00.
After doing a small binary classification on the obtained data set, you'll end up with two classes for each key point, High Number of Buttons Pushed and Low Number of Buttons Pushed.
Now add a listener to your game code and determine when the player enters a new KeyPoint. Use a simple search algorithm to find that if the KeyPoint is in the first class High Number of Buttons Pushed or in the second class, Low Number of Buttons Pushed. If it was in the first class, suggest pressing the key if not pass.
We have an experiment we are running where subjects play a video game and we record various data related to the gameplay. While the game is playing, we make a screen capture of the gameplay. One of the data we capture is when they push a button to request information during the game.
We are trying to understand if there are conditions related to lighting, decision points, etc. that are causing the button to be pushed. Ultimately the goal is to build a predictive model that will present the data before the subject asks for it.
My question is what is the best approach to analyze the video vs. the button push for information? We have timestamped csv files with the button push data and a video of the gameplay. Originally I thought of using Matlab to do the analysis, but I couldn't find a good example of how to approach the problem. I have a couple folks skilled at python on the team that I could ask for help as well, but I had wanted to see if I could understand what needed to be done before tasking them to do it.
Any help is appreciated!
Thanks