 Calthorpe died unmarried at his house on Pall Mall on 11 March 1784 and his remains were interred in the family vault in Ampton church, on 20 March.By his death, the male line of his family became extinct.His relative, Henry Gough (later Lord Calthorpe) inherited his property.