Views: attachment inherits exposed filters when set to No
You can't do that by default with views settings. Filters are meant to stay upon adding / removing one of them, because otherwise you can't search items combining them. You can add a "Reset" button form the "Advanced" block of the view UI (see screenshot).



If you want this behavior to be exactly as you describe, I think you'll need write up a plugin for the filters yourself, or alter the default ones.