Where to configure Time Machine quota when backing up to Mac Server
There was a Time Machine quota setting in macOS Server. Maybe the quota is still set even after the UI has been removed?

Maybe this URL helps:

http://movq.us/2017/04/09/time-machine-quotas/

If I understand this correctly, the quota is set in a hidden plist file in the root of the Time Machine share.

The Time Machine quota setting in "normal" macOS is in the Sharing preferences.
Right-clicking a shared folder has an "Advanced Options…" dialog.



There is also a local Time Machine quota setting but I don't know if this is "forwarded" to network shares, probably not.

https://www.defaults-write.com/setup-a-size-limit-for-time-machine-backup-volumes/

You can check the local quota (if set) via

defaults read /Library/Preferences/com.apple.TimeMachine MaxSize