connect headphone to bluetooth module
Do NOT tie any of the speaker pins of the BT module to ground! This will likely damage the module.

You can wire the speaker and microphone as shown, but as you say, you'll lose half the drive voltage.

As an alternative approach, you could use a 1:1 audio transformer; connect one side to the two speaker terminals of the BT module, and connect the other side between ground and your speaker.