Can you search for only manually added members of a smart group?
I have an experimental search extension that provides many missing features, and includes your needs, I believe.
You can try it if you're up to that https://lab.civicrm.org/artfulrobot/keenies but it should be considered alpha, beta at a push! Please see the README for the searches it can do.
It includes a search on the manual added/removed from a group (smart or not), so using this on a smart group basically ignores the criteria.
I built it because if you're using a smart group for a mailing group, it's helpful to be able to search for people who have unsubscribed, but there could be many uses.
It's in early stages, and may be abandoned if/when the core CiviCRM new search project comes to fruition and provides this functionality (but I built this now because that could be a while).
EDIT: for convenience from the README as of 9 July 2020:

Implemented searches

Has Activity: choose activity types; relationship (with/target, assignee,     created); date range; subject (exact or pattern);
location.

Contact details: contact type(s); created date range; source field.

Manually added/removed from group: select group(s) and a status:    Added/Removed. e.g. find exceptions for smart groups.

In group: select group(s). This is like a normal group search, so works    for smart and non-smart groups. You can put this criteria
in a 'not' wrapper    to find people not in a group.

Has clicked a link in a mailing: date range for the click; select    mailing(s) they must have clicked in.

Has opened a mailing: date range for the open; select    mailing(s) they must have clicked in.

Was sent a mailing: date range and mailing(s) selector.

Has tag(s): select one or more tags.