how can I design Chebyshev filter with 5th-order polynomial and 0.001%-allowed ripples?
A type I Chebyshev filter has a passband magnitude response oscillating between the values $1$ and $1-\delta$, where $\delta$ is the maximum passband approximation error. The Matlab function $\tt{cheby1}$ requires the desired passband ripple in dB:

$$R_p=20\log\left(\frac{1}{1-\delta}\right)$$

The normalized cut-off frequency is

$$\omega_c=2\pi\frac{f_c}{f_s}$$

where $f_c$ is the cut-off frequency in Hertz, and $f_s$ is the sampling frequency in Hertz. The input parameter $\tt{Wc}$ required by $\tt{cheby1}$ is equal to $\omega_c/\pi$.

I don't think that the authors of the paper you refer to got the percentages right. A tip: for a "ripple percentage" of $0.001$ try $\delta=0.0001$.