 Francis's elder brother, Thomas (d.1559), then aged eight, became the ward of their mother's brother, Henry Grey, 3rd Marquess of Dorset, and came to live with Dorset and his wife, Frances Brandon, at Bradgate House, while Francis, then aged two, and his sister, Margaret, were put in the care of Dorset's half-brother, George Medley, the son of Dorset's mother, Margaret Wotton, by her first husband, William Medley.Dorset's ill-fated attempt to place his eldest daughter, Lady Jane Grey, on the throne eventually cost Lady Jane, Dorset, and Dorset's brother, Lord Thomas Grey, their heads, and brought other family members under suspicion.George Medley was arrested, and imprisoned in the Tower for a time, and when he was released in May 1554 was unable any longer to care for Francis and Margaret.