When George Clifford died in 1605, his wife Margaret became dowager countess and began repairing Brougham Castle, which became her favoured residence.
Francis Clifford died in 1641, and the death of his son Henry Clifford, 5th Earl of Cumberland in 1643 left the line without a direct male heir.