Atholl and Governor Smelt resulted in the Duke requesting that Smelt should vacate the Governor's House at Castle Rushen in order that he could reside there whenever he visited the Isle of Man, insisting that Smelt took residence at The George Inn. Smelt himself when writing to the Home Office in 1812 frankly declared he was: The Duke persisted in his harsh treatment of Smelt. On December 7, 1812, the Duke wrote a letter to the Secretary of State for the Home Department, Henry Addington, 1st Viscount Sidmouth, complaining that Smelt: The Duke of Atholl concludes by asking for possession of all public documents as well as the house. Smelt replied to Lord Sidmouth stating that Atholl could use the Courthouse in Douglas, which was deemed better than Castle Rushen, for the Chancery Courts. Lord Sidmouth replied to the Duke, recommending him not to persist with regard to the Governor's House, stating that the Governor's House had been designed by His Majesty's Government as a residence for the Lieutenant Governor. Although the Duke of Atholl persisted in his right to the Governor's House, Smelt was not evicted. In a letter written by Colonel Mark Wilks to Lord Powys in 1812, it was suggested that Smelt was appointed by the Government in order to: Colonel Wilks was voicing the opinion of the House of Keys, of which he was a member, when he declared in further correspondence that: The Attorney General of the Isle of Man was at the time William Frankland, said to of been a relative of Smelt. Frankland declared in a letter to the Home office that Governor Smelt was: In 1816 Frankland was succeeded in the office of Attorney General by John Clarke who, in a letter to the Home Office relative to the constant disputes with the Duke of Atholl, said that: Death Feeling acutely with regard to his health and his position, only seventeen days before his death, on November 11, 1832, Smelt wrote to the Home Secretary asking to be appointed Governor-in-Chief, with a Lieutenant Governor to do the work and permission to stay on in the Governor's House in Castle Rushen. He died in office on 28 November 1832 at Castle Rushen, and was buried under the altar at St Mary's Church in Castletown, his funeral was conducted by the Rector of Ballaugh, the Reverend Hugh Stowell, the oration stating: When St Mary's