Download photos from iCloud Photo Library to an iPhone without uploading everything from this iPhone
iCloud is a syncing service an not a backup service.  

That means that every device logged-in with the same Apple ID will have access to the same files, including photos.  

There's no way to do the selective download that you're asking for.  Either you sync your photos to your iPhone or you continue to selectively download the photos you need.

Other possibilities include not using iCloud for this and using a 3rd party backup service like Dropbox.