 He married secondly Margaret Smyth, daughter of Sir Richard Smythe, 2nd Baronet and after her death in 1631, next Margaret Barkham, daughter of Sir Edward Barkham, 1st Baronet, who died in 1640.By his second and third wife, Irby had no surviving children, three daughters having died as infants.On 19 August 1641, he married finally Catharine Paget, third daughter of William Paget, 4th Baron Paget, and had by her a son and five daughters.