His left-wing Social Democrat party (PSD) had accused him of failing to carry out necessary economic reforms.
The prime minister was involved in a power struggle with party leader Liviu Dragnea, who was barred from office after a conviction for voter fraud.
Mr Grindeanu lost the censure motion by 241 votes to 10.
The decision by his own party and its liberal ALDE allies to oust him was seen as a first, even for Romania's turbulent politics. For the motion to pass they needed a parliamentary majority of 233 votes.
The PSD won elections in December only a year after losing power. Within weeks, protests erupted across the country over a decree seen as weakening anti-corruption measures in one of the European Union's most corrupt member states.
In February the EU warned the government in Bucharest against "backtracking" in its efforts against corruption. Although Mr Grindeanu withdrew the decree, the crisis weakened the new government and soured relations between the prime minister and Mr Dragnea.
Mr Dragnea criticised his rival's performance in government on Wednesday, accusing him of failing to push through "the most ambitious programme since 1989". Some commentators have argued that his removal was more about relaxing measures aimed at tackling corruption.
Mr Dragnea, while being the power behind the left-wing party's throne, cannot take power because he was given a suspended jail term for vote rigging and also faces trial for alleged abuse of office.
His first choice as prime minister, Sevil Shhaideh, was turned down by President Klaus Iohannis in December. A new nominee from the ruling coalition will also have to be approved by the president.
Far from being the sick man of Europe, Romania's economy is showing the highest growth in the EU, at 5.6% in the year to March 2017.
However, its justice system has been under EU scrutiny ever since the country joined in 2007.
In its most recent report this year, the EU's executive praised Romania's anti-corruption agency (DNA) but warned that the fight against corruption was under serious threat from political and media attack.
Romanian Prime Minister Sorin Grindeanu has been forced from power after only six months by his own party, in a no-confidence vote in parliament.