Can somebody be deaf to voices of some people only?
Although you loose the high frequencies in your hearing with age (20kHz and below), this is highly unlikely. The frequency spectrum of our voice is pretty much limited, which is also shown by the fact the for normal telephone, the spectrum transmitted is located between 300 and 3400Hz and the spectrum band between 300 and 3000Hz is named the "voice band". See here for some details.

According to reference 1 the "voice fundamental frequency" for men is between 85 and 180Hz (depending on the reference) and for women between 165 and 255Hz. These frequencies are close enough together to rule out specific hearing of male or female voices. Furthermore the frequencies even overlap.

What happens in age-related pathological hearing loss (also termed Presbycusis) is that voices generally get muffled due to losses in the spectrum and hearing intensity. See the Wikipedia link and also reference 2 for more details.

Reference:


The frequency range of the voice fundamental in the speech of  male
and female adults
Presbycusis