Will AlwaysOn work with hundreds of databases in 1 group/multiple groups?
Increase the hardware capacity or look for Cloud Solutions, if possible. This is from official doc:

Maximum number of availability groups and availability databases per computer: The actual number of databases and availability groups you can put on a computer (VM or physical) depends on the hardware and workload, but there is no enforced limit. Microsoft has tested up to 10 AGs and 100 DBs per physical machine, however this is not a binding limit. Depending on the hardware specification on the server and the workload, you can put a higher number of databases and availability groups on an instance of SQL Server. Signs of overloaded systems can include, but are not limited to, worker thread exhaustion, slow response times for availability group system views and DMVs, and/or stalled dispatcher system dumps. Please make sure to thoroughly test your environment with a production-like workload to ensure it can handle peak workload capacity within your application SLAs. When considering SLAs be sure to consider load under failure conditions as well as expected response times.

I had working experience with huge databases (size), not large in numbers( Availability Groups and databases)
From my experience and based on the research, there is no limit to number of Availability Groups and number of databases by SQL instance or AG. If you have sufficient hardware capacity, you can host any number of databases. If there is not enough memory, even in SQL cluster you will face the issue.