1. An audio signal separation device which generates separate signals by separating each one of plural signals mixed up in plural channels of observation signals in time domain from the observation signals by use of independent component analysis, the audio signal separation device comprising: transformation means for transforming the observation signals in time domain into frequency domain, to generate a spectrogram of the observation signals; separation means for generating spectrograms of the separate signals from the spectrograms of the observation signals; and permutation problem solution means for solving a permutation problem in the spectrograms of the separate signals, wherein the permutation problem solution means calculates a scale corresponding to a degree of permutation from the spectrograms of the separate signals, and exchanges signals at each of frequencies bin of the spectrograms of the separate signals between channels according to the calculated scale by using the plurality of frequency bins for each spectrogram to solve the permutation problem.