Automatically shrink existing Time Machine Backup to a fixed size
Time Machine will respect a size determination - but only when you make the container. This functionality is all that Apple provides for native backup so you'd need a custom setup to do what you ask, and you can't limit things once you've started backing up to a container.

Going forward if you want to make a new restricted container, the concept isn't hard to map out though.


make a new container on the synology and be sure to enforce the size constraints when you start the process. 


Server.app does that easily - I'm not aware of synology code that reverse engineered that feature, but would be surprised if it's not on GitHub or published somewhere public.

At that point, you could delete all backups of user folders that are large and then block by block copy the old backup data to the new destination and then inherit the backup.

You can also thin backup intervals from the command line or script this automation as well.

Lots and lots of work, but it might be worth it as a project if someone is inclined.

When I'm in your shoes, I just copy off the old "too large" backup to an offline hard drive and put it on a shelf and start over with new backups of the size you like. In 6 months or a year - wipe that old drive unless you really feel you need that older history.