Miller was born on November 23, 1823 on a farm near Batesville, Arkansaw Territory to John and Clara Moore Miller.
Miller's political career blossomed upon moving to Batesville, which was the fourth-largest city and one of the most politically prominent cities in Arkansas at the time.