He told Buzzfeed News he needed to go to a "quiet place" to focus on writing, adding that Twitter is "the least quiet place I've ever been in my life."
"It's like taking the bar exam at Coachella," he said. "I really need to concentrate on this!"
Some have labelled his portrayal of female characters, including Scarlett Johansson's Black Widow, as sexist.
The criticism came as a surprise - as the writer-turned-director had previously been praised for creating strong female leads in his TV shows Buffy The Vampire Slayer and Dollhouse.
He said, "I saw a lot of people say, 'Well, the social justice warriors destroyed one of their own!' It's like, nope. That didn't happen."
Whedon also denied the accusation that he quit Twitter because of specific criticism from campaigning group Feminist Frequency.
"Literally the second person to write me to ask if I was okay when I dropped out was [Feminist Frequency founder] Anita [Sarkeesian],"he said.
Immediately prior to deleting his account, Whedon tweeted: "Thank you to all the people who've been so kind and funny and inspiring up in here".
Soon after, a search for an account under the @josswhedon handle returned a "page not found" error.
One critic had created a widely shared montage image showing many of the messages sent to Whedon.
Some called him misogynistic and said he had done a "hatchet job" on Black Widow, whose big storyline in the latest Avengers film is a doomed romance.
One critic asked why the words "strong female character" were not in his vocabulary.
But many of Whedon's fans have now turned on the naysayers, subjecting some to harassment and abuse.
Comedian and actor Patton Oswalt accused them of having "chased Joss Whedon off Twitter".
Whedon had previously been a high-profile user of Twitter, calling it "enormous work - very fun".
But he told Buzzfeed he had started to become uncomfortable with his habit.
"Twitter is an addictive little thing, and if it's there, I gotta check it. When you keep doing something after it stops giving you pleasure, that's kind of rock bottom for an addict.
"I just had a little moment of clarity where I'm like, 'You know what? If I want to get stuff done, I need to not constantly hit this thing for a news item or a joke or some praise, and then be suddenly sad when there's hate and then hate and then hate.'"
Director Joss Whedon has denied leaving Twitter over feminist criticism of his latest movie, Avengers: Age of Ultron.