Matrix has a voltage drop on a coil that should not be receiving any current
If you don't use diodes to steer the current then you will have multiple paths of current flow through all the electromagnets. Yes, there will be one electromagnet that receives a direct connection but, because that row is "grounding" several other electromagnets, current from the active column driver can flow to several series electromagnets and back to the grounded row. Just draw it out and use a crayon or something similar to plot the current flow. Like this: -



The purple electromagnet is the one that is directly driven and the light blue electromagnets are an example of series electromagnets that will also take current flow. I've only shown one path that the extra current will flow but there are many other paths. You need diodes to prevent this from happening.