Prosecutors said he had maintained a "criminal relationship" with officials at a construction conglomerate.
His lawyer said he was innocent and called the arrest a "spectacle".
Dozens of politicians and Petrobras executives have been arrested and sentenced after a two-year anti-corruption investigation.
The investigation, called the "Car Wash operations" has unveiled a massive corruption scheme at Petrobras.
In a news conference, prosecutors said Mr Palocci had favoured construction giant Odebrecht while in government.
They said he had facilitated contracts with Petrobras and helped the group obtain credit from a state bank in exchange for  donations to the governing Workers' Party.
Mr Palocci is alleged to have continued to intermediate bribes even after he left office.
Last week, police temporarily detained another former finance minister, Guido Mantega, also on charges related to the Car Wash probe.
He was released hours later because his wife was about to undergo surgery.
Earlier this month a judge ruled that former President Luiz Inacio Lula da Silva would face trial on corruption charges over the Petrobras scandal.
Mr Palocci served as a finance minister under Lula and as chief of staff to his successor, Ms Rousseff, who was impeached last month for manipulating the budget.
Former Brazilian Finance Minister Antonio Palocci has been arrested in connection with a long-running inquiry into corruption at oil giant Petrobras.