James Madison is most famous for his work on the United States Constitution and the Bill of Rights. He was also president during the War of 1812. Growing Up. James grew up on a tobacco farm in the Colony of Virginia.