How to Remove iOS dev/IDE/... from Xcode?
Delete the Xcode app from /Applications, then delete ~/Library/Developer/Xcode.

You can then install the command line tools separately, by downloading them from Apple's Developer Downloads page (you'll need to log in with your Apple ID). The proper package is listed as "Command Line Tools (OS X version) for Xcode version". Obviously grab the one that applies to your current version of OS X.