Can the rewards be matrices when using DQN?
Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?

Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods.
Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for expected return which would be a weighted sum of matrices, you then get stuck. There is no fixed way to rank matrices and decide whether one is a better result than another. This is a requirement for any learning process that aims to improve at a task - it needs feedback that changes it makes are better or worse related to some reference. As a result, most objective functions and metrics in optimisation use real-valued scalars, which can always be placed into order to decide a highest or lowest value.
This does not prevent you using a matrix representation for your project, if it is a natural fit. To turn it into a usable reward, you will need to convert that matrix into a real-valued metric. Perhaps the L2 norm or other standard measure that summarises the matrix will be good for your task.
It is possible to process multiple scalar rewards at once with single learner, using multi-objective reinforcement learning. Applied to your problem, this would give you access to a matrix of policies, each of which maximised the reward value of one cell within the matrix. It also allows for switching between objectives in a hierarchical manner using a &quot;policy of policies&quot; if some preference for what to achieve changes. It is not 100% clear, but I do not think this is what you want to do.