The letters threatened legal action, but the law firms were false. In some cases Wonga added fees for these letters to customers' accounts.
The City watchdog, the Financial Conduct Authority (FCA), said 45,000 customers would be compensated.
Wonga has apologised and said the tactic ended nearly four years ago.
The City regulator has told the BBC it has sent a file to the police.
The company is the UK's largest payday lender, making nearly four million loans to one million customers in 2012, latest figures show.
An investigation found that Wonga sent letters to customers from fake law firms called "Chainey, D'Amato & Shannon" and "Barker and Lowe Legal Recoveries".
The plan was to make customers in arrears believe that their outstanding debt had been passed to a law firm, with legal action threatened if the debt was not paid.
The company was using this tactic to maximise collections by piling the pressure on customers, the regulator said.
"Wonga's misconduct was very serious because it had the effect of exacerbating an already difficult situation for customers in arrears," said Clive Adamson, director of supervision at the FCA.
"The FCA expects firms to pay particular attention to fair treatment of those who have difficulty in meeting their loan repayments."
The situation occurred between October 2008 and November 2010, and involved Wonga and other companies within its group.
As this happened before the FCA took over the regulation of payday lenders, it is unable to fine Wonga. It also said there would be no criminal investigation as it wanted to set up a compensation scheme as quickly as possible and a criminal probe would take time. Affected customers will receive about £50 each.
Instead, Wonga will start contacting customers in July to offer compensation, with money likely to be paid by the end of the month. This will either be paid in cash or customers will have their outstanding debt reduced.
"We would like to apologise unreservedly to anyone affected by the historical debt collection activity and for any distress caused as a result," said Tim Weller, interim chief executive of Wonga.
"The practice was unacceptable and we voluntarily ceased it nearly four years ago."
Anyone who might have changed address in the intervening period should contact Wonga.
Labour MP and campaigner against payday loans Stella Creasy has questioned the lack of criminal investigation.
"Why in those instances where customers of Wonga charged debt collection fees for these letters is that not police matter?," she asked on social media site Twitter.
Richard Lloyd, executive director of consumer group Which?, said: "It is right the FCA is taking a tougher line on irresponsible lending and it does not get much more irresponsible than this.
"It is a shocking new low for the payday industry that is already dogged by bad practice and Wonga deserves to have the book thrown at it."
The investigation was started by the FCA's predecessor, the Office of Fair Trading (OFT). Wonga said it stopped the tactic voluntarily then offered information to the OFT.
In addition, in April this year, Wonga discovered that it had miscalculated some customers' balances.
This resulted in 200,000 people overpaying the company. Wonga said that the majority overpaid by less than £5, and a larger number underpaid.
Those who overpaid will be contacted by Wonga, and the underpaid debt will be cancelled.
Mr Weller said the company "will learn from these mistakes" and was strengthening its internal controls.
The problems for Wonga come shortly after its boss Niall Wass quit after six months in the job of chief executive. Mr Wass joined Wonga in January 2013 as chief operating officer - after the fake lawyer tactics ended - and became chief executive in November.
Earlier this month, chairman and founder Errol Damelin also announced that he was planning to quit.
Payday lender Wonga must pay £2.6m in compensation after sending letters from non-existent law firms to customers in arrears.