He was selected to fight the South Basildon and East Thurrock seat after ex-Tory MP Neil Hamilton pulled out.
In a recording obtained by the Mail On Sunday, Mr Smith made offensive remarks about gay people, other UKIP members and Chigwell in Essex.
He later issued a "wholehearted and unreserved apology".
UKIP leader Nigel Farage told BBC News that Mr Smith had resigned "by mutual consent", and described his behaviour as "loutish and wholly inappropriate".
He said UKIP had had "great difficulty" with the Basildon selection, adding: "The party has got to grip this and sort it out, in short order".
Mr Farage also said UKIP's national executive committee had the power to impose a candidate if it wanted.
UKIP hopes to make a serious challenge for the South Basildon and East Thurrock seat in the forthcoming general election, in which it is seeking to win a handful of seats and potentially hold the balance of power.
But Mr Smith's resignation, four days after he was re-adopted as a candidate, capped a week of negative headlines for the party.
Mr Hamilton pulled out of contention for the seat amid questions raised by the party over his expenses while another candidate, Natasha Bolter, withdrew amid an investigation into allegations she made against Mr Bird, whose job is to vet election candidates.
Following his resignation on Sunday, Mr Smith said in a statement: "I want the best for South Basildon and Thurrock and I want to see the real issues discussed that touch the lives of people.
"Therefore I have chosen to resign so that UKIP can win this seat next May."
A parliamentary candidate resigns having tried blaming his racist comments on taking painkillers.
This comes days after an alleged sex scandal at UKIP Head Office in which the party's chief executive did - or did not - sleep with another candidate.
Meantime a wealthy donor is said to be threatening to stop funding the party if his friend doesn't get a seat. You may think that UKIP's week of bad headlines is just a diverting soap opera.
You may think it simply shows the growing pains of Britain's fastest growing political force. You may think it has no significance at all. If so, you'd be wrong.
Read Nick's full blog
During the recorded conversation, Mr Smith talks about UKIP's lesbian, gay, bisexual and transgender (LGBT) group. He can be heard jokingly referring to it as BLT UKIP, and adds "what the old poofter groups call themselves".
He jokes about "shooting peasants" from the Essex town of Chigwell and supporting "a peasant's hunt through Chigwell village".
A UKIP spokesman confirmed to the BBC that Mr Smith had apologised to Mr Farage for allegations made against him during the phone call - which Mr Smith has since retracted.
UKIP MEP Steven Woolfe, the party's immigration spokesman, said that "racist and homophobic language" would not be tolerated.
"When there is someone who has come out and expressed these type of views, which I abhor and dislike intensely, we have acted rapidly and we have got them out," he told Radio 4's Today programme.
UKIP's candidates, he claimed, were "ordinary people" who did not have the media training that their political rivals had and sometimes had to be "guided".
But he insisted the way the party was dealing with problems as they arose showed how it had matured.
"What we have seen over the last ten days is that UKIP has grown up rather dramatically," he added.
Mr Smith stood for UKIP in the Basildon seat in 2010, where he came fourth behind the Conservatives, Labour and the Lib Dems.
But the party now regards the seat as one that it could possibly win and the selection process is now likely to be re-opened.
After Mr Hamilton withdrew his candidacy for the Basildon seat earlier this week, Channel 4 reported seeing a letter from UKIP's finance committee querying expenses claims made by him.
Mr Hamilton, UKIP's deputy chairman, said his decision to withdraw from the Basildon selection process was nothing to do with expenses, but that it was a UKIP "dirty trick" to stitch him up.
Kerry Smith has resigned as a UKIP prospective parliamentary candidate after apologising for offensive remarks he made in a phone call.