Pixel Wise Write into a Video File in MATLAB
I assume your matrix is 3D where the first 2 dimensions are Width and Height and the third is time (Gray Scale Video).

If you can write your processing as a Filter you can use MATLAB's filter() function.
The function has the dim property hence you can set it to 3 and it will apply on the filter over the third dimension.
The limitation is applying the same filter to whole data.
If you need to have different filter for each pixel then you should iterate over the filter bank.

Another solution is to use the functions permute() and reshape() to rearrange the data to have the Time Axis in the dimension which fits your needs.