Raikkonen was 0.109 seconds clear of his team-mate, with Rosberg 0.237secs off the pace as Ferrari hinted they may offer a threat to Mercedes in Shanghai.
Hamilton, with a five-place grid penalty as a result of a gearbox change, was 0.433secs off the pace.
Red Bull's Daniel Ricciardo was fifth ahead of Toro Rosso's Max Verstappen.
Defending champion Hamilton, who trails Rosberg by 17 points in the standings, said it had been "a difficult day" and said it had been "quite windy" on the track.
He added: "Ferraris are particularly strong, so I think we're going to have a fight this weekend.
"It seems like we're going to have a bigger fight than we've ever had, so it'll be good for the fans."
Double world champion Fernando Alonso, who has been cleared to race after missing the last grand prix in Bahrain on medical grounds, was 11th quickest, just ahead of McLaren team-mate Jenson Button.
Sunday's Chinese Grand Prix - the third of the season - is live on the Sport website and radio 5 live at 07:00 BST.
There was no repeat of the spectacular left-rear failures that afflicted Williams driver Felipe Massa twice and Renault's Kevin Magnussen in the first session.
Massa was 14th after the team apparently solved the two wheel failures suffered by the Brazilian in the first practice session.
Williams head of performance Rob Smedley said the team were "fairly confident" they knew what the problem was but refused to go into details.
Valtteri Bottas, who was 10th, said the team had found a problem with the "rim clearance" - the distance between the inside of the wheel and the wheel assembly.
Magnussen also had a problem with the left rear of his car in the first session.
That turned out to be a suspension failure and he did not run in the second session as the team worked on a solution.
Ferrari's pace in heading the Mercedes in the timesheets might appear to suggest that they can challenge McLaren this weekend, but Ferrari have flattered to deceive already this season.
Vettel, for example, was fastest in final practice at the last race in Bahrain, only for the Mercedes cars to go half a second quicker than the Ferraris in qualifying.
"If we get everything right, we can potentially be a bit closer," said Vettel. "But it is only Friday, so I wouldn't put too much importance in the times."
Headline lap times can be misleading as teams can manipulate the fuel loads, while the times set on the race-simulation runs on heavier fuel loads can present a more accurate picture.
On these, Mercedes had a clear pace advantage, although Raikkonen appeared able to make the super-soft tyre on which the top teams will start the race last longer at a stronger pace than Rosberg and Hamilton.
Chinese Grand Prix Second practice results
Chinese Grand Prix coverage details
Ferrari's Kimi Raikkonen and Sebastian Vettel topped second practice at the Chinese Grand Prix, ahead of Mercedes duo Nico Rosberg and Lewis Hamilton.