How can I accomplish the Iron Man in helmet effect around the head with out a black or green screen?
You can try this and see if it gets you close to where you want to be:

It's going to be subjective, so play with the Mix Fac to see if that does the trick for you.





ALTERNATIVELY You could start doing something like this as well.
For the right transition, you may have to layer a few more iterations of things before you nail it ( Somthing Like this ).



!!Appended Since Comments Below!!

Here's an example of layering some of these effects (It May be a poor one, but it should give you a little more perspective). Also I've attached the blend so you can look at the setup a little closer.