Filtering/Searching fields with enumerated data (e.g. Smart Map) in an ElementCriteria model
UPDATE
As of Smart Map 2.3.0, this is now possible!

https://plugins.doublesecretagency.com/smart-map/filtering-entries-by-subfield-value/


ORIGINAL
Currently, this isn't possible. However, it's a popular feature request which is rising quickly on the priority list.
The current workaround (as you noted) is to simply filter the results afterward.