Her paternal grandparents were William Leslie, 3rd Earl of Rothes and Janet Balfour, and her maternal grandparents were William Crichton, 3rd Lord Crichton of Auchingoul and his mistress, Princess Margaret Stewart, youngest daughter of King James II of Scotland and Mary of Guelders.
Agnes's father died in 1558 in Dieppe, France while on a diplomatic mission.