What is the meaning of h(r) in this protocol?
As Ricky Demer already mentioned, $h$ is a hash function. 

In the realms of cryptography, it mostly points to a secure hash function. But in the end, it doesn’t matter if it’s a secure hash or not… $h$ is standard notation for “hash”.