The Serbs lost 6-4 6-4 to Brazilians Marcelo Melo and Bruno Soares in Rio.
The loss came a day after Djokovic was knocked out of the singles by Argentine Juan Martin del Potro.
World number one Djokovic, 29, suffered a surprise third-round defeat at Wimbledon in July when he lost to American Sam Querrey.
The 12-time Grand Slam winner came back to claim his 30th Masters title in Toronto before the Olympics.
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
Novak Djokovic lost in Rio for the second time as he and 40-year-old team-mate Nenad Zimonjic exited the men's Olympic doubles in round two.