Pattern recognition for temporal data
Are you trying to detect speech vs non-speech, or are there classes of non-speech sounds you are trying to discriminate from? I'm not clear from your question.

I think a decent first approach would be to block your signal into frames and compute the Mel-Frequency Cepstral Coefficients (MFCCs), as well as the delta-MFCCs (differences between adjacent frames' MFCCs) and delta-delta MFCCs (differences between MFCCs in frames that are two frames apart). This isn't the only way to do it, but without more specific knowledge of the problem domain, this is probably a good place to start.

Just googling should give you some good reference on how to compute the MFCCs if you are not already familiar with them. Basically you take the DFT, take the magnitudes, compute the energies inside triangular windows corresponding to human hearing, take the DCT of these coefficients, essentially as a compression step, and then discard the high order coefficients, usually only taking about the first twelve coefficients. I have an explanation of the meaning of the DCT step in this post: How do I interpret the DCT step in the MFCC extraction process?

You could then, say, use these coefficients as features for an SVM.