How is the ends of a bandpass filtered signal reliable after applying the cosine taper? A taper function decaying to zero at ends such as a cosine window is routinely applied to the signal before bandpass filtering to reduce end effects. Obviously the window alters the waveform. 

So, as for the filtered waveform, how is it reliable near the beginning and end? 

how does the the decay ratio of the window (e.g. 5%, 10% ...) influence? It's selected arbitrarily or by some criteria?

In the filtering functions of matlab and scipy, some extrapolation is applied to the ends of the signal to match their slope. In such situations, is the windowing still necessary?
A taper function is applied for spectral estimation if the artifacts from a non-tapered rectangular window provide less useful results that the artifacts due to the tapered window.

Not tapering a finite length sub-sample vector is equivalent to rectangular windowing, which convolves Sinc shaped artifacts onto the spectrum.