Born at the family estate called "Mount Custis" near Drummondtown, Virginia, Bayly attended the common schools as a child and went on to study law at the University of Virginia, graduating in 1829.
In 1844, he left that post when he was elected as a Democrat to fill a vacancy in the United States House of Representatives.