How to scale-down in a multi-tenant environment?
Autoscaling is a good case for machine learning

This is a hard problem to do well.

What you really want is something like Nest Thermostat for your EC2 infrastructure.

There are (aforementioned) multiple dimensions of resource demand/limitations.


CPU
memory
disk space
disk IO
network IO
concurrency/latency/queue depth


There are multiple indicators of demand (in addition to the above).


concurrent unique visits/sessions
pages per visit/session
rate of engagement/interaction feature usage


There are multiple common patterns of changing demand over time.


daily user demand cycles
weekly user demand cycles
monthly ...
annual ...
special event/days
DDoS load
media/marketing exposure traffic spikes


There are multiple financial decision factors.


does revenue scale with traffic? How? (How conservative do you need to be?)
is there a hidden cost to control (transaction costs, limits)?
what's the cost model of scaling? (things in a pipeline scale together, things in a load-balanced cluster scale independently)


Before long, if you try to hand-optimize on hand-selected features you're going to have a monster of technical debt that is possibly more complicated than any other logic in your site. Amazon makes more money when you err (with a large margin) on the side of caution, so their tools will probably never get close to what you want.

Instead, choose an architecture/technology stack that can grow/scale so you don't have to get it exactly right the first time. Then pick a few factors which you think are obvious. Then try to come up with a way to sort multiple representative possibilities in order of preference. Then collect some real world data covering all those points. If you're lucky, a simple obvious hand-coded solution will jump out at you from looking at the data. If not, code up something that will give you an approximate model f(x1,x2,x3,x4) --> y * app nodes, using an appropriate algorithm.

I bet you didn't think this one was going to be so much fun!