The 32-year-old made the last of his 63 Test appearances in the 16-12 victory over Italy in Turin last month.
Hamilton made his Scotland debut against Romania in November 2006.
"I have loved every single moment of wearing the thistle and feel extremely humble to have done it for nine years," the Saracens player said on Twitter.
"Today is both a sad day and a day that I feel extremely proud to have had.
"Thank you to the Scotland supporters for making it so special."
Hamilton was one of 11 players to miss out when Cotter cut his 42-man training squad to his final selection on Tuesday.
John Barclay, Blair Cowan, Adam Ashe, Hugh Blake, Chris Cusiter, Alex Dunbar, Rob Harley, Ruaridh Jackson, Duncan Taylor and Greig Tonks were also omitted.
Scotland open their World Cup Pool B campaign against Japan in Gloucester on 23 September.
They then face United States in Leeds before meeting South Africa and then Samoa in Newcastle.
Scotland lock Jim Hamilton has announced his retirement from international rugby after he was left out of Vern Cotter's World Cup squad.