While 7% of people underpay and could be fined, 13% are paying too much.
The Office of Rail and Road (ORR) wants train companies to refund passengers who accidentally buy tickets that are too expensive for their journey.
Train operators said that vending machines were designed for simple ticket requirements.
They say plans to simplify fares will help them to give clearer information on ticket machines.
The ORR employed mystery shoppers to buy rail tickets from machines.
"Despite investment in new technology and the removal of jargon from ticket machines, our new research shows passengers may be paying more for their journey than necessary," said John Larkinson, ORR director of railway markets and economics.
"We are calling on train companies to commit to refund anyone who finds that they could have bought a cheaper ticket for the same journey," he added.
Rail operators C2C and Scotrail already have a price guarantee in place for when passengers overpay.
Earlier this month, the Rail Delivery Group (RDG), which represents train operators, announced a trial scheme to simplify fares.
It said the 16 million fares on offer were "baffling" for passengers.
As part of the trial, which begins in May, ticket machines on stations will also be updated, to make sure they show the cheapest fares available.
"To quickly benefit passengers, the Rail Delivery Group must set out what improvements to ticket machines will be made in the short term," said Mr Larkinson.
Paul Plummer, chief executive of the RDG, told the BBC that operators were aware that there was a problem with machines, and that they had formulated a 10-point plan to improve them.
Yet he pointed out the machines were designed as "queue-busters" for passengers with relatively simple requirements, but "expectations [of passengers] had evolved".
He said that the system of fares was far too complicated.
The RDG also added: "Complex, decades-old government rail fare regulations make it more difficult for train companies to offer the right, simple options on ticket machines.
"Simplifying the number and types of fares in the system will let train companies provide customers with clear information and help them to make better informed choices," they added.
by Ben Morris, business reporter
In the British Rail era, rail fares were a relatively simple affair, with ticket prices worked out on a cost-per mile basis.
But in the 1990s, in an effort to make the network more efficient, the operation of rail services was handed to private train operating companies (Tocs).
The Tocs were given some freedom to set ticket prices, with advanced ticket bookings particularly important to them, as it helps them to keep their trains full.
To protect passengers, other fares were regulated, including season tickets and off-peak fares.
The result is a patchwork of fares, that becomes even more complicated when a passenger wants to cross several train regions.
Savvy travellers have been taking advantage of this by splitting their journey into several tickets, rather than one through-ticket.
Rail ticket machines cause so much confusion that one-fifth of passengers who use them buy the wrong ticket, according to the rail regulator.