Bluetooth Speaker Controls Too Loud and in Too Few Steps
After a couple of hours of searching I found the fix:


Enable Developer options if you haven't yet (instructions can be found online).
Open Developer options.
Enable the Disable absolute volume option.
Enjoy listening to music at whatever volume you like.


Here are the details for this option: