Why are microcontroller WiFi modules expensive compared to USB WiFi adapters?
That is because they are not the same. Arduino/Launchpad/Hobbyist target boards are aimed at small microcontroller sales, which does not have a lot of volume, so price breaks are minimal. And microcontrollers often need more memory than is available for some of the things. A mass produced device can narrowly tailor it's processor to the task at hand.

But one of the biggest things, is that the wifi (or ethernet) chips aimed at hobbyists is that they incorporate more than just the communication protocol. Some also handle the tcp/ip stack, which is very memory intensive and hard to code. Some allow for simple serial (uart or spi or i2c) communication between the microcontroller and the wifi chip, while it does the rest. You are paying for added value.

And finally, most of the time, hobbyists are buying completed boards, not raw chips. Some of these wifi chips, like the TI CC3000 require reflowing, because they have pins underneath or bga packages. They are not easily hand soldered.