From September, NHS staff and council workers will be among those required to have language skills equivalent to GCSE grade C or above.
Managers will have to test employees can "communicate effectively with the public".
Cabinet minister Matt Hancock said it was to help control immigration.
The new rules, which will form part of the government's immigration bill, will mean every public sector worker in a "customer-facing" role will have to be able to speak at least school-leaver levels of English.
It will include police officers, social workers, teaching staff and assistants and council employees.
The government wants the policy to be rolled out across England, Scotland and Wales and said it would be working with the Scottish and Welsh governments on the details of the plans.
Each organisation will have to decide how to test their staff based on guidance in a code of practice, which the government will produce after a consultation starting in the autumn.
Existing employees who are not fluent enough may be given time to improve.
Doctors working in the UK are already required to have a much higher level of English and are tested by the General Medical Council.
The risk of a healthcare professional not being fluent was highlighted by a mistake made by Dr Daniel Ubani, a German doctor who gave a lethal dose of a painkiller to patient David Gray in 2008.
At the time, as a European citizen, he was able to register to work in the UK without passing a language test but rules were changed in June last year.
Mr Hancock said: "We are controlling immigration for the benefit of all hard-working people.
"We have already introduced tough new language requirements for migrants, now we will introduce new legislation in the forthcoming immigration bill to deliver the commitment made by the prime minister to go further."
People who do not speak fluent English will be barred from public sector jobs which involve working directly with the public, the government has said.