In 1805, Brown was defeated for re-election to the Senate and retired to Liberty Hall.
In 1812 Brown was appointed by the legislature to oversee the construction of a public house of worship on the public square of Frankfort.