Can I disable speakerphone on HomePod?
I suppose you've got two devices to play with in terms of the settings, your HomePod and the iPhone.

The first thing to check is your call routing settings on your iPhone:


Go to Settings > General > Accessibility
Tap on the Call Audio Routing option
Try changing these to see if you can achieve what you want


You can also try restricting it with a password, but this may not be an ideal solution in your case and may end up being worse. However, it's worth a try to see how it'll work in terms of the speakerphone function in case having a password changes the default behaviour:


Launch the Home app
Tap on the small arrow at the top right corner
Tap on Home Settings
Tap on Home
Now enable the Allow Speaker Access option (under the Speakers heading)
Finally, enable the Require Password option.


Now test this to see if it's a practical solution in your case.