Disable Carrier Phone Number for Google Voice Number
Since Google Voice just comes in as a normal carrier call, there is no simple way to separate those calls. It doesn't pass any identifying information that the phone could use to identify it is a Google Voice call versus a carrier call. 

You can, however, change your Google Voice settings (on the website, not the app) so that it always shows your Google Voice number as the caller ID on incoming calls. This means you could just whitelist your Google Voice number and block everything else. Then you will receive only GV calls, but you will lose the ability to see who is calling since all calls coming to GV will just display as your GV number. You could then use the call screening features of GV to identify unknown callers. 

It really isn't an elegant solution and the loss of caller ID is a big inconvenience, but if you absolutely have to do this, it is about your only option.