Simplest way of detecting where audio envelopes start and stop
What you really want to do is essentially called as Voice Activity Detection or speech detection. 

Basically any pure speech signal (which contains no music) has three parts. 


The voiced sound - which is basically caused by Vowels 
The unvoiced sound - which contains consonants. 


The characteristic of human sound is such that while a lot of energy is used in voiced sound the real information is contained in consonants. Also, voiced sound is usually lower frequency where as unvoiced sounds are higher frequencies. [To be precise all voiced sound are resonated more or less a constant frequency for a given person which is his/her pitch].

Now, as any system there is noise. The voiced sound is usually quite powerful enough that it can be distinguished visible. When you apply a lower frequency filtering it is possible to collect good magnitude of voiced sounds however, the unvoiced sound (with all the rich information) will get lost. 

Coming to the question how to solve it:  

The trick lies in the fact that unvoiced sound still come from a resonating source; and inherently restricted over a certain frequency. Where as, the noise is rather uniform. So a simple measure that distinguish all three is "local power" or alternatively but equivalent is to take the windowed auto-correlation. 

If you take at a time say 100 samples - and auto correlate itself, if it contains only noise the results will be pretty much zero (this is the property of white noise) where as for speech signal, this magnitude will be observable because the signal still has better structure.  This has worked for me in the past. 

VAD has been an active research areas- because almost all Mobile phone communications wants to detect non speech part and remove them from encoding. But if they would remove non-voiced speech this would make telephony useless. 

The G.729 standard computes VAD based on features like: line spectral frequencies, full-band energy, low-band energy (&lt;1 kHz), and zero-crossing rate.

The GSM standard works as follows: Option 1 computes the SNR in nine bands and applies a threshold to these values. Option 2 calculates different parameters: channel power, voice metrics, and noise power. It then thresholds the voice metrics using a threshold that varies according to the estimated SNR. (from wikipedia)

For more advanced techniques i am listing some references on this subject. 


Most sited reference: Jongseo Sohn;   Nam Soo Kim;   Wonyong Sung;  "A statistical model-based voice activity detection" Signal Processing Letters, IEEE, Jan 1999, Volume: 6 Issue:1 pp:1-3 
Most relevant for you: Mark Marzinzik and Birger Kollmeier "Speech Pause Detection for Noise Spectrum Estimation by Tracking Power Envelope Dynamics" IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 10, NO. 2, FEBRUARY 2002 pp.109 
Ramírez, J.; J. M. Górriz, J. C. Segura (2007). "Voice Activity Detection. Fundamentals and Speech Recognition System Robustness". In M. Grimm and K. Kroschel. Robust Speech Recognition and Understanding. pp. 1–22. ISBN 978-3-902613-08-0.
Introductory : Jonathan Kola, Carol Espy-Wilson and Tarun Pruthi "Voice Activity Detection"