Fine Bluetooth Volume Control
Found the answer:
To decouple phone volume and speaker volume enable what's now called Disable absolute volume in developer options (looks like used to be called Disable max volume)
Taken from here