Strictly limit macOS Photos library size using "Optimize Storage"?
Limiting Photos iCloud Photos Library sizes was discussed on Accidental Tech Podcast:
https://atp.fm/334 (1:36:45)
https://atp.fm/335 (0:04:20)
https://atp.fm/410 (2:19:05)
They originally suggested using a Sparse Disk Image with a size quota.
Follow-up suggested instead using an APFS Space-Sharing volume.

Hey, others may have said this, but don’t use a sparse image for your photo library. Add an APFS volume with a 30GB quota instead. Much, much safer! (twitter)


APFS Space-Sharing Volume
In Disk Utility, select Add APFS Volume..

APFS volumes share storage space within their container.


Set a Quota Size to strictly limit the size of the new volume:

Size Options
APFS Volumes share storage space within their container. The optional reserve size ensures that amount of storage will remain available for this volume. The optional quota size limits how much storage this volume can allocate.


When you move your Photos library, your iCloud photos will only be permitted to fill this Quota Size set for the volume.

Move Photos Library
Finally, move your Photos library file to the new size-limited volume.
Quit Photos.app &gt; Move the library file &gt; Hold Option and relaunch Photos &gt; Pick the library’s new location &gt; Photos Preferences &gt; Use as System Photo Library
How to put Photos.app main library on an external disk?
Or, start a new Photos library on the volume and allow iCloud to re-download your photos.