Do swept back wings affect the moment of inertia of an aircraft enough to be relevant for the roll response?
A wing has three basic design parameters—span, chord and sweep—which are all independent of each other.

Span is always measured along the perpendicular axis of the whole aircraft and chord is always measured along the longitudinal axis. So pivoting the wing around its root will change all three parameters and affect the properties that depend on them.

When designing the aircraft, you choose area (span times chord) to give you the needed amount of lift, then span to give you the desired minimum lift-to-drag speed and sweep for transsonic capability and/or stability. So you should be comparing swept and unswept wing of the same span and area, and those somewhat obviously have the same moment of inertia in roll.