The Planning Inspectorate has backed the council's decision to demand the property near Crymych is demolished.
Owner's Charlie Hague and Megan Williams argued that the straw-walled house had low impact on the environment.
But inspectors said that did not change rules on building in rural areas.
"There is a lack of proper justification for the benefits of the low-impact development in this case for this matter to be given sufficient weight and to outweigh the policies which seek to control development in the countryside," ruled the planning inspector, Iwan Lloyd.
The owners had been living in a caravan for four years, before moving into the new home just before their son Eli was born.
Sculptor Mr Hague spent more than a year creating the single-storey dwelling from local materials.
The couple were helped by friends and relatives at the neighbouring Lammas eco village.
But Pembrokeshire council began enforcement proceedings earlier this year, after ruling it was an unauthorised dwelling built in open countryside.
Deputy council leader Rob Lewis said he welcomed the planning inspector's decision.
"Planning is a rigorous process with clear guidelines and if it is to be enforced fairly must also be upheld by everyone," he said.
The enforcement notice gives the couple two months to demolish the property.
However, the council will now consider a retrospective planning application for the house.
"It will be considered on its merits," said Mr Lewis.
A Pembrokeshire couple who built a 'hobbit-style' eco home without planning consent have lost a demolition appeal.