Is a ground plane meaningful even where the signal return path is not through ground?
I don't see any need for the ground plane in the matrix area. It could serve some purpose for reducing noise, but likely there is no measurable difference. 

A strip under the USB is probably the best you can do in your case. Make it at least one trace width wider than the diff pair, the wider the better.

Going for 4 layers would be the best choice, but if you need to save those pennies, you can make it work with 2 layers.