How do you calculate a unit cell's dimensions from an XRD spectra?
You should not really have assumed that the compound was tetragonal. If it was indeed tetragonal, then the peaks corresponding to the $(200)$ and $(020)$ planes would overlap. Instead, you can see that one occurs at $2\theta = 46.7^\circ$ and the other at $2\theta = 47.6^\circ$. The question may be considered a little misleading, but it does hint at this; it says that the compound is "nearly tetragonal", and it gives you the general formula for an orthorhombic crystal, which is one with $a \neq b \neq c \,(\neq a)$ and $\alpha = \beta = \gamma = 90^\circ$.

The next problem is that you are given values of $2\theta$ (which happens to be the convention), but the formula reads $\sin^2{\theta}$. As such, you need to make the appropriate adjustments.

This question is relatively easy because the peaks have already been indexed for you (i.e. the values of $h$, $k$, and $l$ are already found out). Therefore, all that is left is to plug the numbers into the formula you are given. Which peaks you use is entirely up to you, although it is obviously easier to use $(h00)$, $(0k0)$ and $(00l)$ peaks.

The reflections of the $(200)$ plane occur at $2\theta = 47.6^\circ$. So:

$$\begin{align}
\theta &amp;= 23.8^\circ \\
\sin^2{(23.8^\circ)} &amp;= \frac{(1.54~Å)^2}{4} \cdot \frac{2^2}{a^2} \\
a &amp;= 3.81~Å
\end{align}$$

Similarly, you can show using the $(020)$ and $(003)$ peaks that $b = 3.89~Å$ and $c = 11.64~Å$. This is actually pretty good. The reported values of $a$, $b$ and $c$ are $3.8231~Å$, $3.8864~Å$, and $11.6807~Å$ respectively.

Regarding units, if you make sure that you leave your units inside the intermediate workings (as they should be), you should never run into confusion about "what unit should the answer be". In your question, you converted $\lambda$ into units of $\mathrm{nm}$. $\sin^2\theta$, $h$, $k$, and $l$ are all dimensionless. Therefore, your calculated values of $a$, $b$, and $c$ must match the units of $\lambda$ you used, i.e. $\mathrm{nm}$.

In my answer, I used ångströms throughout. It is not wrong to use $\mathrm{nm}$, but as Babounet noted in the comment section, ångströms are more commonly used in crystallography.