Edward was the tenth child, and youngest son, of Richard Woodville, 1st Earl Rivers and Jacquetta of Luxembourg, who came to prominence when their daughter Elizabeth Woodville married King Edward IV.
In 1472 he accompanied his brother, Anthony Woodville, 2nd Earl Rivers, to Brittany.