Envelope reconstruction algorithm for audio processing I need to reconstruct the envelope of a sound. 

Audio data are professionally-recorded natural sounds (speech, bird songs) with very little noise. I would prefer working in the time domain rather than in the frequency domain (I've seen some algorithms based on FFT transformations that looked overcomplicated for what I need). The algorithm will be implemented in an interpreted language so it needs to stay "light" in computation.

As a first approach, I considered using a peak detection algorithm, then doing a linear interpolation between the peaks. But isn't there some pitfalls with such a naive approach? Are there some standard ways of implementing envelope reconstruction in the time domain that would better suit my needs?

FWIW, I'm not familiar with digital signal processing vocabulary, so do not hesitate to reword my question if I misused some terms
I would recommend a streaming RMS detector. The standard approach for computing a streaming RMS detector is to square the input samples and then apply these to a 1st-order lowpass filter. If you want the output in dB, take 10*log10() of this quantity. If you want the output in volts, take the square root of this quantity. If Logs and square-roots are too heavy for your processor, there are low-mips ways to approximate them.
Note that RMS detection offers a good compromise between a peak detector (which is often over-responsive to very narrow transients) and average (which is sometimes too slow). Also, the ear tends to approximate RMS dynamics in its perception of loudness.