Phipps married Catherine Sawyer, daughter of George Sawyer, and granddaughter of Sir Robert Sawyer who was Attorney General to Charles II, counsel for the defence at the Trial of the Seven Bishops in the reign of James II and Speaker of the House of Commons.
William married Lady Catherine Annesley, daughter of James Annesley, 3rd Earl of Anglesey; she was a granddaughter of James II.