Smart group with manually added contacts
You should think of a smart group as two parts:


A static group (and all the tables it involves)
A query and its associated tables


The final list of contacts is the union of the above two (minus the contacts who have been explicitly removed from the static group)

I think the above design allows a lot of flexibility and power. I think the main issue with smart groups is being smart about when and how to cache the results. Civi does a fairly basic job of this right now and can be improved considerably (IMO)