 By another warrant by Grafton dated 16 February 1742, he was appointed Yeoman of the Removing Wardrobe, an office he held until it was abolished in 1782.Calthorpe first came to reside at his family's ancestral home, Ampton Hall, in 1736, and immediately set about improving his mansion and estate by enlarging the former, and enclosing, planting, and ornamenting the latter; dividing his time in attendance on his official duties in London, and in agricultural and horticultural pursuits, when resident in the country.In 1754, Calthorpe offered himself as a candidate for the borough of Hindon in Wiltshire, but declined, although sure of his election, in favour of James Dawkins.