Harry John Neville Vane, the 11th Baron Barnard, died at Raby Castle near Staindrop on Sunday. He was 92.
In a short statement, his family said he died peacefully surrounded by loved ones.
Lord Barnard, who succeeded his father in 1964, held numerous titles in County Durham. He is survived by five children.
The Raby estate covers large parts of Teesdale including numerous farms and houses which can be recognised by their whitewashed walls.
Raby Castle staff tweeted: "So much thanks for the messages of condolence we've received. Lord Barnard will be sadly missed."
The Vane family have held the Barnard title since 1698.
Click here for more on this story and other news from across the North East
Lord Barnard has died at his County Durham castle, his family has confirmed.