iPhone 4S restore from backup 64 GB to 16 GB
This is not possible within iTunes, which deals with backup and restore as a single block. Keep in mind that syncing apps is different from a backup and restore - so you could remove some apps to free up space and try the restore.

The backup is stored under ~/Library/Application Support/MobileSync/Backup/, but the files not in an easy-to-use form.

If you wish to explore your backup and see what can be selectively extracted or used, there are some paid applications to do that (check the free trials to see if they're helpful):
iBackup Extractor
iPhone Backup Extractor

Also see:
Processing iPhone / iPod Touch Backup Files on a Computer