Installing extension as non superuser on Amazon AWS RDS I would like to allow non super users to install a specific postgres extension. Accordingly to the documentation at https://www.postgresql.org/docs/current/extend-extensions.html that is possible if I set superuser to false  in the extension .control file.

However, in AWS RDS that is not possible, because I don't have access to the filesystem. Would there be a way to set those .control parameters via SQL query? Or did I reach the usability limit that RDS offers me?
Amazon RDS has a custom list of allowed extensions. You can see it with:
SHOW rds.extensions;

See online documentation.
And you should do it as role rds_superuser or as a member of that role (actual superuser is not available). Their website informs:

The rds_superuser role can do the following:

Add extensions that are available for use with Amazon RDS. [...]