1. A signal processing device for processing a plurality of observed signals at a plurality of frequencies, the plurality of the observed signals being produced by a plurality of sound receiving devices which receive a mixture of a plurality of sounds, the signal processing device comprising: a storage unit that stores observed data of the plurality of the observed signals, the observed data representing a time series of magnitude of each frequency in each of the plurality of the observed signals; an index calculation unit that calculates an index value from the observed data for each of the plurality of the frequencies, the index value indicating significance of learning of a separation matrix using the observed data of each frequency, the separation matrix being used for separation of the plurality of the sounds; a frequency selection unit that selects at least one frequency from the plurality of the frequencies according to the index value of each frequency calculated by the index calculation unit; and a learning processing unit that determines the separation matrix by learning with a given initial separation matrix using the observed data of the frequency selected by the frequency selection unit among the plurality of the observed data stored in the storage unit.