The businessman, from Essex, denied ill-treating his wife, who is in her late 70s, during a High Court hearing.
The court's Family Division heard the woman was frightened of her husband and had taken legal action against him.
Judge Catriona Murfitt concluded the woman was likely to suffer "significant harm" if he stayed in the house.
The details of the case emerged after the man was ordered to vacate the property but went on to appeal the decision, saying the ruling was "unfair".
Mr Justice Baker dismissed the challenge, telling him he had "no prospect" of overturning Judge Murfitt's ruling.
He said the couple, who have two adult sons, could not be identified.
In his ruling, Mr Justice Baker said the woman claimed "that for the duration of the marriage, she had been the emotional punch bag for his insecurities and frustrations".
She also said he had been physically abusive towards her on occasion, and that he was an "extremely difficult" man who had been unfaithful and "disinhibited in his behaviour".
Judge Murfitt had made an order requiring the businessman to "vacate the property altogether", Mr Justice Baker said.
"She held that the wife was likely to suffer significant harm if the order was not made and that the harm that she would suffer was greater than any harm which the husband was likely to suffer," he said.
A 67-year-old millionaire has been ordered by a judge to leave his home after his wife said she had been used as "an emotional punch bag".