Is there a way to use decimation or linear interpolation to shrink or stretch an audio signal in the time domain?
If you are resampling by an integer factor you can simply decimate the audio.
If you want to down sample directly in the time domain you can apply interpolation. One type of interpolation is the linear interpolation, but you have others, MATLAB provides interpolation out of the box with the interp1 function.
If you have sampled wave data, you can separate each channel you can get the resampled data as
resampled = interp1(y, 1:interval:length(y), smoothing);

The interval is the fined as (1 + speedup) * Fs / Ft where Fs is the sampling frequency and Ft is the target sampling frequency, and speedup makes the playback faster or slower (same if speedup=0)
Here an example use of interp1
y = randn(1, 10);
t = linspace(1, 10, 1000);
hold all;
plot(t, interp1(t, y, 'linear'))
plot(t, interp1(t, y, 'linear'))
plot(t, interp1(y, t, 'linear'))
plot(t, interp1(y, t, 'cubic'))
plot(y, 'or')
l = legend('linear', 'cubic', 'spline')
set(l, 'fontsize', 12)


The same is possible in python using scipy.interpolate functions. But in that case you first create a sampler and then you call it how many times you want.