What happens when the amount given doesn't allow for equilbrium to be achieved in a reaction?
This takes the assumption that the water vapour present acts as an ideal gas. Therefore $PV=nRT$ is valid.

Flaws/Misconceptions:
(1) You directly used pressure as you have in the RICE table.

Using pressure in a RICE table is not very useful since the RICE table deals with the amount of reactants and products and not pressure.

(2) The reaction doesn't reach equilibrium

The reaction reaches equilibrium as shown below. This means that the question in title doesn't need to be considered.
However, in such a case where equilibrium isn't reached, then the reaction stops due to lack of reagents to go forward. When you add more reagent it keeps moving forward until equilibrium is reached after which addition of solid reactants wouldn't cause a reaction to take place.


Now, as OP has stated, the $P_\ce{H2O}= \pu{20 mm Hg}$ at equilibrium. and it is given that relative humidity is $80\%$ for a saturation vapor pressure of $\pu{85 mm Hg}$ and so that means the intial pressure exerted by $\ce{H2O(g)}$ is:
$$P_\ce{H2O}= \frac{8}{10}\cdot 85=\pu{68 mm Hg}$$
Now using the ideal gas law, for the initial condition, we get:
\begin{align}
n_i(\ce{H2O})&amp;= \frac{68}{76}\cdot\frac{1}{0.0821 \times 300} \\
&amp;\approx\pu{0.00036 mol}
\end{align}
Similarly for the final case, we get:
\begin{align}
n_f(\ce{H2O})&amp;= \frac{20}{76}\cdot\frac{1}{0.0821 \times 300} \\
&amp;\approx\pu{0.00011 mol}
\end{align}
Therefore, $\pu{0.00025 mol}$ of $\ce{H2O}$ reacts for the condition to reach equilibrium. Now drawing the RICE table for the reaction we get:
\begin{array}{lc} \hline
\ce{&amp;A.3H2O &amp;&lt;=&gt; &amp;A.H2O &amp;+ &amp;2H2O(g)} \\ \hline
i &amp;1\times10^{-3} &amp;&amp; 1 \times10^{-3} &amp;&amp; 3.6 \times 10^{-4} \\
c &amp;1.25\times10^{-4} &amp;&amp; -1.25\times10^{-4} &amp;&amp; -2.5\times10^{-4} \\
e &amp;1.125\times10^{-3} &amp;&amp; 8.75\times 10^{-4} &amp;&amp; 1.1\times10^{-4} \\ \hline
\end{array}
So the final composition is:
\begin{array}{lc} \hline
\text{Component} &amp;\text{Amount} \\ \hline
\ce{A.3H2O} &amp; \pu{1.125 \times10^{-3} mol} \\
\ce{A.H2O} &amp; \pu{0.875 \times10^{-3} mol} \\
\ce{H2O} &amp; \pu{1.1 \times10^{-4} mol} \\ \hline
\end{array}