Is there a policy that can limit the growth of an AWS S3 buckets total size?
Not as far as I know. 

You can create Billing alerts to get notifications when AWS thinks that your monthly spend may be over a given threshold.