The social messaging service's co-founder Jack Dorsey will take over as interim chief on 1 July and stay until a replacement can be found.
Mr Costolo had been under pressure from investors unhappy with the firm's user growth.
In a statement, he said he was "tremendously proud of the Twitter team".
Twitter said that its board had formed a committee to undertake the task of finding a successor.
Following the announcement, shares in the company jumped by more than 7% in trading after US markets had closed.
Twitter debuted on the New York Stock Exchange in 2013 but has not been adding users as fast as investors had hoped.
In April, the firm missed Wall Street's forecasts for revenue growth and posted a net loss of $162m (Â£104m).
Its share price has declined nearly 30% since then, and is currently trading below the price it debuted at in 2013.
Analysts remain pessimistic about the firm's ability to grow.
The research consultancy eMarketer estimates that Twitter's monthly user base will grow just 14.1% this year, slowing from more than 30% two years ago.
By 2019, eMarketer says Twitter's user growth rate will be just 6% worldwide.
In announcing his resignation, Mr Costolo said in a statement: "I am tremendously proud of the Twitter team and all that the team has accomplished together during my six years with the company.
On a conference call to discuss the move, he added that he had decided to go now because he felt the continued scrutiny if he remained would be a "distraction" and of "no help to the company".
Questioned on what the board was looking for in its next chief executive, Mr Dorsey said the search team would take as long as necessary and the candidate could come from inside or outside the company. But he added that they knew one thing for sure: he or she should be a Twitter user.
Mr Dorsey said the most important attribute was that they "really use and love the product in every single way".
The problem Costolo couldn't solve was how to attract new users.
Or, at least, enough new users to keep growth-hungry Wall Street investors happy.
And of the people that did sign up, analysts say they weren't being given enough reasons to stick around for very long.
Why did Twitter boss stand down?
Nate Elliott, an analyst at Forrester research, said the news of his departure wasn't surprising.
"Twitter has never been great at giving its users reasons to come back," said Mr Elliott.
"While other social sites have introduced new features and functionality the past few years, Twitter has mostly stood still. The result has been excruciatingly slow user growth."
However, some have cautioned that lofty investor expectations regarding Twitter's ability to turn in to the next Facebook might be overblown.
Technology blogger John Gruber wrote on the site Daring Fireball: "I think what Wall Street wants is a pipe dream: for Twitter to turn into another Facebook.
"No [chief executive] is going to make that happen."
Mr Costolo will remain on the board of the company. He has been a director since September 2009 and became chief executive in October 2010.
Dick Costolo is stepping down as chief executive of Twitter, the company has announced.