Removing vocal but retaining all background ambient noises
What you are trying to archive is very hard if at all possible. The pinnacle of research today (including my own) is striving for good results on such tasks. DCASE challanges engage with similar tasks with some interesting results, though the make assumption which cannot be generalized to your case. In DCASE2019 task 3, for example, they assume up to 2 timewise parallel sources where in your case there are no such limitations. 

With respect to your conditions and constraints, I would advise looking into source separation and\or classification. If your background audio has a repetitive nature, this paper may help.