How many people have worn the straw hat, if only for a second?
Here you can read who had worn the straw hat and what it has gone through during the series. I will summarize the list here and add proof from the manga. For the complete panel the thumbnails are cut from, click the thumbnails. Until chapter 801,  it seems 7 people were seen wearing the hat.




Chapter 2 − Monkey D. Luffy


  Luffy is the current holder of the straw hat.






Chapter 1 − Shanks


  He gave it to Luffy as a young boy after he said the same words that Gold Roger, his former captain, had once said.






Chapter 603 − Gol D. Roger


  Much later in the series, it was revealed that Gold Roger, the man who would become the pirate king, had owned that hat.






Chapter 81 − Nami


  She is the member of Luffy's crew that he more often gives it over to for save keeping.






Chapter 114 − Robin
Not mentioned on the list, but as explained by OP, Robin had taken Luffy's hat from him during their first encounter.





Chapter 314 − Usopp


  Usopp briefly wore the straw hat during the Davy Back Fight against the Foxy Pirates.






Chapter 515 − Sweet Pea


  Luffy was in a panic when he couldn't find his hat, but it saw one of the women in the crown watching him wearing hit.