Relation between the virial coefficients and van der Waals constants
You have to bring both equations into the same form and then compare the coefficients. You seem not to have used the tip with the Taylor expansion and I guess that's why you got an incorrect result.

As you wrote: The virial equation of state is

\begin{align}
p &amp;=\frac{nRT}{V} \left(1+\frac{nB}{V}+\frac{n^2C}{V^2}+ \ldots \right) \\
  &amp;= \frac{RT}{V_{\mathrm{m}}} \left(1+ B\frac{1}{V_{\mathrm{m}}}+ C\frac{1}{V_{\mathrm{m}}^2}+ \ldots \right)
\end{align}

and the van der Waals equation is

\begin{align}
 p &amp;=\frac{RT(1-\frac{b}{V_{\mathrm m}})^{-1}}{V_{\mathrm m}}- \frac{a}{V_{\mathrm m}^2} \ .
\end{align}

Now, you have to bring the van der Waals equation into the same form as the virial equation by factoring out $\frac{RT}{V_{\mathrm{m}}}$

\begin{align}
 p &amp;= \frac{RT}{V_{\mathrm{m}}}\left( \left(1 - \frac{b}{V_{\mathrm{m}}} \right)^{-1} - \frac{a}{R T V_{\mathrm{m}}} \right) \ ,
\end{align}

using the Taylor expansion $\left(1 - \frac{b}{V_{\mathrm{m}}} \right)^{-1} = 1 + \frac{b}{V_{\mathrm{m}}} + \frac{b^2}{V_{\mathrm{m}}^2} + \ldots$ and ordering the terms according to their power in $\frac{1}{V_{\mathrm{m}}}$

\begin{align}
 p &amp;= \frac{RT}{V_{\mathrm{m}}}\left( 1 + \left(b - \frac{a}{R T} \right) \frac{1}{V_{\mathrm{m}}} + b^{2} \frac{1}{V_{\mathrm{m}}^{2}} + \ldots \right) \ ,
\end{align}

so that you can compare the coefficients in both equations to get the result

\begin{align}
 B &amp;= b - \frac{a}{R T} \\
 C &amp;= b^2 \ .
\end{align}