There are growing divisions in the right-wing party over Mr Orban's steps to turn Hungary into an "illiberal democracy", despite a third election victory this year:
Even though a majority of Hungarians oppose their policies - 2.8 million against to 2.3 million in favour in April -  opposition parties are weak and divided, with the exception of the radical nationalist Jobbik movement, which has emerged as the main rival to Fidesz.
But the mood in the corridors of power is wretched. In a country where backbench rebellions are almost unknown, politicians contend themselves with hints of dissent in anti-government media, and more open attacks on Viktor Orban's policies in pro-government media.
A new, pro-Russian foreign policy sits particularly awkwardly in a party which rose to fame in 1989 as the first to demand publicly that Soviet troops leave Hungary.
It began with a photograph of Viktor Orban shaking hands with Russian President Vladimir Putin in January, after signing a highly controversial nuclear expansion deal.
When the Russians annexed Crimea in March, the Budapest government criticised the beleaguered Ukrainian government in Kiev, and stated publicly that all it cared about was the Hungarian minority in western Ukraine.
In July, Hungary's embassy in Estonia was closed down at a moment when the Baltic states - Hungary's Nato allies - felt under most pressure from Moscow.
Mr Orban has consistently opposed EU and US sanctions against Russia, on the grounds that "we should be doves in economic policy, and hawks in defence".
In an August speech to Hungary's ambassadors, he called openly for an unprincipled foreign policy, placing foreign investment in Hungary above any moral considerations such as a trade partner's human rights record.
And in September, within an hour of Mr Orban meeting with the chief of Russia's state-owned gas giant, Gazprom, the reverse flow of gas to Ukraine in Hungary's pipelines, which had been annoying Russia, was stopped.
Dismay is growing within Fidesz at his domestic policy too: at government corruption, the playboy lifestyle of numerous party officials, and an economic policy which consistently suppresses the value of the Hungarian currency, the forint.
There is also increasing rivalry between the oligarchs: the rich men who fund the party from behind the scenes, on a model learnt from the former communists.
"Unprincipled decisions cause harm, even if that is not visible in the short term," parliament speaker Laszlo Kover, a founding member of Fidesz and close friend of Mr Orban said in August in the main pro-government weekly newspaper.
This was in reply to a question on the rise of communist-era secret service loyalists and businessmen with offshore accounts in government ranks.
The editor of the same weekly, Gabor Borokai, himself a former Fidesz government spokesman, lambasted what he called a "self-assured and arrogant social group" of young Fidesz leaders who believed they were beyond the law. This was apparently a reference to 36-year-old Foreign Minister Peter Szijjarto and his friends.
Another intra-Fidesz dispute has broken out over the extensive power of Janos Lazar, 39, the minister in charge of Mr Orban's office, who is responsible for the daily running of the government.
Other Fidesz officials, led by the head of the parliamentary group, increasingly regard Mr Lazar as a reckless and dangerous figure.
"From Hungary to Egypt, endless regulations and overt intimidation increasingly target civil society," US President Barack Obama said in September. This followed a government investigation into non-governmental organisations partially-funded by Norway, over alleged financial mismanagement.
The latest insult to split Fidesz came last week when the US government banned six unnamed Hungarian public officials from visiting the United States.
While the most hysterical pro-government commentators called for revenge, investigative reporters suggested that the head of the Hungarian tax authority was on the list for trying to bribe US businessmen.
That was followed this week by anger at a new government plan to tax internet use.
"Orban's illiberal democracy is the expression of a power structure where all powerâ€¦ resides at the centre," says Peter Rona, an Oxford-based economist and former member of the supervisory board of the Hungarian National Bank.
Unlike almost all other commentators, however, he can see an end to Fidesz's rule.
Hungarian governments are traditionally brought down by the misguided policies of their central bankers.
And Mr Rona thinks Fidesz could fall too because the determination of national bank head Gyorgy Matolcsy, a former Orban minister, to keep the forint weak at all costs.
Hungarians are commemorating the anniversary of their 1956 revolution against Soviet rule but many senior figures in the ruling Fidesz party believe Prime Minister Viktor Orban is steering the country back to its authoritarian past.