How can I train a many-to-one RNN with an array of 2D matrices?
you can use CNN just before inputting your raw matrix into RNN. An other option, which is just made for that purpose is ConvLSTM. Several research papers show that ConvLSTM can be used to process in sequential frames of video which is similar to your input type.