One of the men was arrested in the capital Berlin and the other in a town close to Cologne, in west Germany.
Police seized computers, mobile phones and sketches in the raids but did not find a "smoking gun", prosecutors' spokesman Martin Steltner said.
About 450 officers were involved in the raids.
One of the men detained, reported to be aged 35, had been living in a refugee shelter in the town of Attendorn, east of Cologne, and is wanted by the Algerian authorities for alleged links to IS.
Police said "investigations show that he has been trained militarily in Syria".
The man's wife, also wanted by Algeria, was detained at the same time but prosecutors said she was not a suspect in the German case.
The suspect held in Berlin was arrested on suspicion of falsifying identity documents, police said.
Two other Algerians in Germany, said by police to be "from the jihadist scene", were tracked down but not arrested.
"Our understanding is that the four men accused could have planned to carry out an attack together," Mr Steltner said.
German media said the possible targets included central Berlin tourist sites Checkpoint Charlie and Alexanderplatz.
Since January, the suspects had changed their mobile phones multiple times and communicated using instant messaging services, reports said.
Germany's security concerns have risen since the Paris attacks last year.
Officials declined to comment on a report in the Tagesspiegel newspaper, citing security sources, which said senior IS figures had ordered an attack on Germany.
German police have arrested two Algerians suspected of planning an attack and having links to the militant group, Islamic State (IS).