Dolley Payne Todd Madison ( May 20 , 1768 â `` July 12 , 1849 ) was the wife of US President James Madison . James Madison was president from 1809 to 1817 while Dolley Madison was the First Lady of the United States . Dolley Payne Todd Madison ( May 20 , 1768 -- July 12 , 1849 ) was the spouse of the fourth President of the United States , James Madison , and was First Lady of the United States from 1809 to 1817 .