If he wins the support of voters he will be the first elected UK Independence Party MP in the Commons.
The maverick Eurosceptic backbencher said he did not believe Prime Minister David Cameron was "serious about the change we need" in Europe.
The PM described his defection as "regrettable" and "counterproductive".
Mr Carswell said the decision to jump ship from the Conservatives had given him "sleepless nights" but he wanted to see "fundamental change in British politics" and UKIP -  a party he believed belonged to its members rather than a "little clique" of political insiders - could deliver it.
"This hasn't been an easy decision. I have been a member of the Conservative Party all my adult life. It's full of wonderful people who want the best for Britain.
"My local Conservative Association in Clacton is thriving. It brims with those I'm honoured to call my friends.
"The problem is that many of those at the top of the Conservative Party are simply not on our side. They aren't serious about the change that Britain so desperately needs.
"Of course they talk the talk before elections. They say what they feel they must say to get our support... but on so many issues - on modernising our politics, on the recall of MPs, on controlling our borders on less government, on bank reform, on cutting public debt, on an EU referendum - they never actually make it happen."
He said only UKIP could "shake up that cosy little clique called Westminster".
On Mr Cameron's pledge of an in/out EU referendum in 2017, after renegotiating powers back from Brussels, he said the prime minister's advisers had "made it clear that they're looking to cut a deal that gives them just enough to persuade enough voters to vote to stay in".
He added: "Once I realised that, my position in the Conservative Party became untenable."
Mr Cameron - who was not warned by Mr Carswell about his plan to defect - said: "It's obviously deeply regrettable when things happen like this, when people behave in this way.
"But it's also, in my view, counterproductive. If you want a referendum on Britain's future in the EU - whether we should stay or go - the only way to get that is to have a Conservative government after the next election.
"And that is what until very recently Douglas Carswell himself was saying."
Speaking exclusively to BBC political editor Nick Robinson in Glasgow, the prime minister said the by-election in Clacton would be held "as soon as possible" and he "wants to make sure there's a very strong Conservative campaign in that seat".
"I want to go early to Clacton for this reason: people in Clacton voted not just for Douglas Carswell, they voted for a Conservative government, for a Conservative member of Parliament," he added.
Analysis by Brian Wheeler and Iain Watson
Douglas Carswell's announcement took Westminster by complete surprise. UKIP had told everyone they were about to unveil a major new celebrity donor.
No-one expected a Conservative defector to be unveiled instead. The identity of the defector is, perhaps, less of a surprise - Mr Carswell has a long history of rebelling against his party and is known to be deeply disillusioned with what he sees as David Cameron's failure to clean up Westminster politics and deliver democratic reforms.
But it is Mr Carswell's decision to call a by-election in Clacton - rather than continuing to sit in the Commons under a different party banner - that will cause the biggest headache for Mr Cameron.
The prospect of defeat is real. And if it happens, expect more Conservative MPs to call for a pact - at least informally, seat by seat - with UKIP at next year's general election.
It will encourage Eurosceptic MPs to harden their rhetoric on a referendum and renegotiation. And for swing voters worried about the cost of living, they will hear the Conservatives - to use the prime minister's words - "banging on about Europe".
Technically the seat could be left vacant until the general election. But Mr Cameron cannot look like he is "running scared" on Europe.
In South Thanet the Conservatives selected a senior UKIP defector.  They will have to trawl their membership lists to find another Eurosceptic to take on Mr Carswell.
Conservative MP Mark Pritchard - a fellow Eurosceptic - said Mr Carswell had been "flirting with UKIP for some time" and his defection would inflict "short-term" damage to the Conservative Party, as he may win the by-election.
But he said voters would realise a "vote for UKIP is a vote for Labour".
UKIP leader Nigel Farage, who this week was selected to fight South Thanet at the 2015 general election, shared the platform with Mr Carswell at his press conference.
Tory Douglas Carswell has defected to UKIP and quit as MP for Clacton, saying he will contest the subsequent by-election for Nigel Farage's party.