He was the great-grandson of Sir William Ayloffe, 1st Baronet, through his third wife (Alice, daughter of James Stokes of Stoke near Coventry), their first son was Joseph Ayloffe, of Gray's inn.
Sir Joseph seems very early in life to have manifested an interest in antiquities, which received at once the recognition of the learned, although for many years he was merely collecting information and published nothing.