I want Glass to be transparent where nothing exists
One way you could do this is by using a sort of "greenscreen".


Add a big sphere which encloses the entire scene, and give it a material which will only be visible when seen directly through a transmissive material. Also disable visibility for unwanted ray types:


Put the sphere on it's own layer (I put it on layer 2), and create a second renderlayer which excludes it:



The first renderlayer is identical except for the Exclude setting.
Use the compositor to turn the green channel of one renderlayer into the alpha channel of the other:








If there is already red green and blue in your scene, you could try using the compositor to turn the difference between the two renders into transparency:





However you may need to tweak things to get it to work as desired.