How to check if audio samples have only noise or are silent? I have a wav file I want to split into frames in order to feed it into a machine learning model. The problem is that the audio has silence with some noise at some points.

My problem is that I do not want to include frames with no sound (or only noise) in my dataset. 

One solution I believe is to use a model for speech recognition or something similar to do classification and see if a frame includes only silence or noise.

However, I am searching for a solution that will not rely on machine learning but mostly on signal processing techniques or some other pre-processing method.

So, how could I exclude these frames with only silence or noise?
First you should apply speech enhancement algorithms to remove noise from speech. Then, you should use VAD (Voice Activity Detector) to remove silence from speech.