How can I tell whether my friend is using iMessage or Message to SMS me?
If the messages are grey then they are being received. The colour doesn't change on the type they use. However if you use iMessage then the message back will be iMessage unless it stats above it that it is a message, as both users need to have iMessage enabled at the same time to use the feature, otherwise an SMS message is sent.

If you scroll up in the message history you will see what the last type of message used was. There will be a line break with iMessage or Message written in it. All messages after that point are dove via this method, until a new break is made.