 Madison sought to step down from his duties as bishop in 1805; whether an assistant bishop was chosen, as he appears to have wished, is somewhat unclear.He remained as President of the College of William and Mary for 35 years, serving until his death, although quite infirm in his last years and unable to attend the General Convention.Bishop Madison died on March 6, 1812, about a year after he ordained future bishop William Meade as a deacon.