Can you customize the install of Xcode from the Mac App Store?
Don't accept this answer without vetting it further first, but I think I noticed something during my install.

I think that you don't get any SDKs as part of the XCode 4 install. It was only after I installed XCode 4 and launched it, that the project window showed a download in progress. Clicking it gave me the option of downloading: OSX 10.6 SDK, Legacy OSX SDK, iOS SDK, and something else that I don't remember at the moment. Perhaps extended documentation.

So, the XCode install itself is still very heavy. What with XCode visually, all it's associated applications and libraries, Clang/LLVM/LLDB, profiling tools, and everything.

But, the SDKs themself? I don't think it comes automatically. I can't seem to figure out how to get back to choose the other SDKs to download, so I'm not 100% sure in this.