1. An audio signal processing apparatus comprising: an audio acquisition unit configured to acquire an audio signal including a voice signal and a non-voice signal; a likelihood calculation unit configured to extract audio features expressing features of the voice signal and the non-voice signal from the audio signal, and calculating likelihood expressing probability that the voice signal is included in the audio signal; a spectral feature extraction unit configured to perform a frequency analysis to the audio signal to extract a spectral feature; a first basis matrix producing unit configured to produce a first basis matrix expressing the feature of the non-voice signal using the spectral feature; a second basis matrix producing unit configured to specify a component having a high association with the voice signal in the first basis matrix using the likelihood, and excluding the component to produce a second basis matrix; and a spectral feature estimation unit configured to estimate a spectral feature of the voice signal or a spectral feature of the non-voice signal performing nonnegative matrix factorization to the spectral feature using the second basis matrix.