At the time, it was the custom to have a geographic balance on the Supreme Court, and Baldwin had been from Pennsylvania. Polk's efforts to fill Baldwin's seat became embroiled in Pennsylvania politics and the efforts of factional leaders to secure the lucrative post of Collector of Customs for the Port of Philadelphia. As Polk attempted to find his way through the minefield of Pennsylvania politics, a second position on the high court became vacant with the death, in September 1845, of Justice Joseph Story; his replacement was expected to come from his native New England. Because Story's death had occurred while the Senate was not in session, Polk was able to make a recess appointment, choosing Senator Levi Woodbury of New Hampshire, and when the Senate reconvened in December 1845, Woodbury was confirmed. Polk's initial nominee for Baldwin's seat, George W. Woodward, was rejected by the Senate in January 1846, in large part due to the opposition of Buchanan and Pennsylvania Senator Simon Cameron.[233][234]
Despite Polk's anger at Buchanan, he eventually offered the Secretary of State the seat, but Buchanan, after some indecision, turned it down. Polk subsequently nominated Robert Cooper Grier of Pittsburgh, who won confirmation.