Fully Remove Google Voice Search
Open your Google app (not chrome). 
Press "more" 
On the new screen press "settings"
You should see "google assistant" and "voice", press "voice"
Press "voice match" under the"ok Google" section. 
You have finally reached the screen you need. Here turn of the setting "access with voice match"