[165][166] Due to the opposition of Monroe and Clinton, Madison immediately faced opposition to his planned nomination of Secretary of the Treasury Gallatin as secretary of state. Madison eventually chose not to nominate Gallatin, keeping him in the treasury department.[167]
Madison settled instead for Robert Smith, the brother of Maryland Senator Samuel Smith, to be the secretary of state.[166] However, for the next two years, Madison performed most of the job of the secretary of state due to Smith's incompetence. After bitter intra-party contention, Madison finally replaced Smith with Monroe in April 1811.[168][169] With a Cabinet full of those he distrusted, Madison rarely called Cabinet meetings and instead frequently consulted with Gallatin alone.[170] Early in his presidency, Madison sought to continue Jefferson's policies of low taxes and a reduction of the national debt.[171] In 1811, Congress allowed the charter of the First Bank of the United States to lapse after Madison declined to take a strong stance on the issue.