Correlation between 2 signals of uneven dimensions As a part of my work, I am trying to correlate the audio signal in a video with the pixels of each frame. The steps I follow are:


Audio sampling rate and frame rate of the video are known. So, division of the audio sampling rate with the frame rate gives us the number of audio samples associated with each frame of the video.
That being done, I want to correlate the audio signal (the collection of audio signal amplitudes with 2 channels) with the 2-D pixel intensity of each frame, to basically decipher whether which part of the frame has maximum correspondence with the audio.
But, obviously the 2 signals ( audio signal with 2 channels and the 2-D frame pixel intensity values ) are having different dimensions. In this scenario, how do I compute the correlation between them?


I did encounter the idea of performing canonical correlation, but then again, the unequal dimensions of the 2 signals makes it a problem.
The intuitive way to go about this would be to consider every single pixel in your video (assuming that's only intensity, not e.g. color) a 1D signal over time.

Then you'd get width×height number of crosscorrelation functions between pixel intensities and your audio.

The different sampling rates of audio and video just mean that you'd need to reduce your audio to a signal of a rate that is identical to the video frame rate. (You could also interpolate the video to the audio rate, but then you'd be mostly making statements about the correlation properties of your interpolation function, not of your video).

The question is how you can reduce your audio to something that only takes say 25 values in a second. My wild guess is that this is something where you either use your inherent ideas of what would be a clever measure (e.g. "loudness in 40ms steps") or you just do the same for all 12.5 Hz wide bandwidths of your audio, and see which correlates best.