Media playback is not supported on this device
The 29-year-old scored a single yuko to beat Graf 1-0 in the first bronze medal match in the weight category.
Conway had earlier beaten world champion Gevrise Emane of France but was beaten by Colombia's Yuri Alvear for the right to fight for gold.
The Bristol-born judoka won bronze for Scotland at the 2014 Commonwealth Games.
"It hasn't sunk in," she told BBC Sport. "I knew if I was going to come away with a medal I would have to put in my best performances.
"I'm not from Brazil but the crowd have been awesome. I just want to thank everyone."
Find out how to get into judo with our special guide.
Conway suffered shoulder ligament damage during her second-round defeat at London 2012 and had to have surgery.
"This has been my dream since London," she added. "I can't put it into words."
Great Britain's Gemma Gibbons, who won judo silver at London 2012, tweeted: "She's only gone and done it.
"Absolutely amazing from Sally Conway. Our Olympic medallist Miss Conway."
Japan's Haruka Tachimoto beat Colombia's Yuri Alvear by ippon to win the gold medal and Germany's Laura Vargas Koch defeated Spain's Maria Bernabeu by wazari to win the other bronze medal contest.
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
Briton Sally Conway defeated Austria's Bernadette Graf to win the -70kg judo bronze medal.