It will consider the deployment of police officers in undercover roles by police forces in England and Wales.
It comes after controversy surrounding the conduct of undercover officers.
Meanwhile, a separate review into the relationship between the Home Office and the SDS says a key file is missing.
The inquiry announced on Thursday will also make recommendations about how undercover policing is conducted and will scrutinise the use of undercover officers by the now-defunct SDS and National Public Order Intelligence Unit.
Among the allegations are that undercover officers took fake identities from dead children, had relationships with campaigners and fathered children.
Mrs May said that the work of barrister Mark Ellison and Operation Herne - an investigation into undercover policing at the Metropolitan Police - has unearthed "serious historical failings" in undercover policing practices.
Mr Ellison - who in 2012 successfully prosecuted Gary Dobson and David Norris for the murder of teenager Stephen Lawrence - has been examining police corruption during the original investigation into the killing in 1993.
She added: "As I said last year, I was profoundly shocked by Mark Ellison's findings and committed to establishing a public inquiry to thoroughly investigate undercover policing and the operation of the SDS.
"While I initially said that Mark Ellison's further work and criminal investigations needed to conclude before the inquiry commences, it has become apparent that these pieces of work were much larger than initially envisaged.
"In the interest of learning the lessons of past failures, I have decided to establish the inquiry now while ensuring existing work is not affected.
"Undercover policing is an essential tactic in the fight against crime but to improve the public's confidence in undercover work we must ensure there is no repeat of these failings in the future."
This latest inquiry will be led by judge Lord Justice Pitchford.
Mrs May had announced her intention to set up a judge-led public inquiry following the report of the independent Ellison Inquiry in March last year.
That had found Scotland Yard spied on Stephen Lawrence's family.
And Operation Herne found that the SDS had carried out undercover operations to gather information on 18 justice campaigns over 35 years.
This included the campaign regarding Jean Charles de Menezes, shot by police after being mistaken for a terrorist in the days following the London Underground bombings in 2005.
The SDS was set up in 1968 to infiltrate pressure groups and remained operational until 2008.
It has been the focus of controversy over claims that some undercover officers had sexual relationships and fathered children with campaigners, to help build trust in their false identities.
Chief Constable Mick Creedon of the Association of Chief Police Officers said: "Undercover policing is a vital police tactic has saved lives, protected victims and brought dangerous criminals to justice."
He also said that the public inquiry would help make sure that the "unacceptable behaviour of some officers in the past never happens again".
"Changes have already been made to the authorisation and management of these operations.
"New training and guidance for undercover officers guidance developed with the College of Policing reinforces the need for them to meet the highest ethical standards."
The allegations about the SDS include that its officers worked to discover information about the Lawrence family, which could be used to smear them following Stephen Lawrence's death.
His father, Neville, said he would like to be fully involved in the inquiry.
He added: "I've always felt that my family were under greater investigation than those guilty of killing my beloved Stephen and so it is only natural to want to discover what was really going on."
Jules Carey, of Bindmans solicitors, which represents the woman known as "Jacqui" who had a child with an undercover officer, said: "For the inquiry to fulfil its purpose of restoring public confidence it will be vital to ensure that there is proper consultation with all interested parties in the framing of the terms of reference for the inquiry."
She also said that the inquiry must be "conducted in an open manner that allows those affected by spying to question the undercover officers and those who responsible for their deployment".
Meanwhile, Mrs May has also published an independent review of the Home Office's knowledge of SDS activities, commissioned in March last year and conducted by Stephen Taylor, former director at the Audit Commission.
Mr Taylor said it was not possible to say if a missing file regarding SDS, which would have included documents classified as "secret" and "top secret", was due to "human error or deliberate concealment".
Mr Taylor also said that "given the absence of documents and the passage of time" he could not rule out the possibility that "an individual or individuals within the Home Office" may have been aware of:
An inquiry into undercover policing and the operation of the Metropolitan Police's Special Demonstration Squad is being established by Home Secretary Theresa May.