Handling Phase Shift with ANC
These are the design requirements of a good speakerphone to eliminate the sidetone from the mic.  But in a speakerphone, the latency from mike to the speaker is a short delay and wavelength so the phase shift in the mic and speaker is fairly easy to measure and compensate with something to get 180deg.

But not in your case.

With non-flat phase response mic and speaker with wall reflections so it MUST be a noise cancelling mic that will not pickup speaker sounds. This means with feedback there will always be a frequency with sufficient feedback gain to cause it to howl or oscillate.  Then you have two choices, I think. 


Use two cheap mics in differential phase mode such that your voice is in between the two mics. The low-frequency response is cut off due to the wavelength between the mics, unless it is offset so you speak in one and use the other to cancel far-field sounds without disturbing the feedback to both. THen the gain is controlled by the drain resistor which may need to be tuned.
A cheap passive mic noise cancellation with breathing mesh cloth on the back.
The may or may not balance of pressure on the mic depending on the random quality of electret or MEMs mics. 


The catch is it only picks up near-field acoustic pressure, with your voice near the corner of your mouth with a steady position on a headset.