(feature discontinued) Google Hangouts phone app: How to start a group SMS where one of the recipients is a Hangouts user?
You can do this in two different ways:

Method 1:


In Hangouts, tap the + button 
Tap "New SMS"
Find Mary and tap on her entry.  The combination of step 2 and 3 will solidify the message as an SMS.  If you select Joe (our Hangouts contact) first, it may convert to a Hangouts message, so it is important to start with your SMS contact first!  
Find Joe and tap on his entry
Tap the green arrow to continue.  In my case, I have two numbers I can send from (my carrier number or a Google Voice Number), so I would select my desired number.


Method 2:


In Hangouts, send a message to Mary
While staying in Mary's conversation, tap the "three dots" button in the upper right
Tap "New Group MMS"
Find Joe and tap on his entry
Tap the green arrow to continue.  In my case, I have two numbers I can send from (my carrier number or a Google Voice Number), so I would select my desired number.


Testing

In either situation, you can test if you were successful by tapping the "three dots" button in the upper right of your group message.


If the dropdown has an option for "Delete", then you are in a group SMS/MMS
If the dropdown has an option for "Leave", then you are in a hangout


Tested on Hangouts version 10.0