Does driving a speaker with a DC offset AC signal matter?
Yes.



Figure 1. Cutaway view of a dynamic loudspeaker for the bass register.
1.    Magnet,
2.    Voicecoil,
3.    Suspension,
4.    Diaphragm. Source: Wikipedia.

A DC component in the signal will cause:


Bias of the speaker cone (4) and the suspension (3) from the neutral position. This means that the cone will reach the limit of its travel - or at least where it becomes highly non-linear - in one direction in a shorter distance compared with the other.
Maximum undistorted volume will be decreased as a result.
The coil (2) will be heating even when no audio is playing.