Why can't a balance that can weigh to the nearest milligram be able to weigh 23.8 milligrams of something?
Couldn't agree less, but check my math.  

First $\ce{C18H38}$ has a molecular weight of $254.5~\mathrm{g/mol}$. 

Second $\mathrm{5.64\times 10^{18} ÷ 6.02214 \times 10^{23} = 9.3654 \times 10^{-6}}$ and so is $2.4\ \mathrm{mg}$.  

A balance with an ability to weigh to the nearest mg would, in a perfect world give you a mass of 2 or possibly 3 mg. This wouldn't be very useful, but it wouldn't give you zero information, either. So, since I don't understand the context in which your book asked the question, I'd have to say it doesn't seem to be completely correct if it states "it cannot" weigh that amount. It certainly cannot weigh that amount with adequate precision for most purposes I can think of. But if, say, you were trying to determine whether there was enough of a sample to use in a mass spec., or to calculate the yield of a reactor-on-a-chip, then it would suffice.