How do I need to configure my site on AWS with Elastic Load Balancing to force HTTPS everywhere?
Got it myself.

Configuration is absolutly correct, but i set the wrong reverse_proxy_adresses to be allowed. I thought, that i can use the beanstalk-url, but it worked perfectly fine with one of the internal IP Configs for AWS: 172.16.0.0/12