James Duddridge, who has tabled a no confidence motion in Mr Bercow, said he had used his job to "pontificate" on international issues and had spoken publicly against Brexit.
The Speaker, who recently criticised US President Donald Trump, told students on 3 February he had voted Remain.
Lib Dem chief whip Tom Brake said Mr Bercow was "even-handed" in his job.
And the Speaker's spokeswoman said he had been "scrupulous in ensuring that both sides of the argument are always heard".
But speaking on BBC Radio 4's Today programme, Mr Duddridge said: "Enough is enough.
"We need a new more impartial Speaker."
Although Mr Duddridge is currently the only signatory to his motion, he said he had received calls from ministers and MPs from different parties saying they too would vote to oppose Mr Bercow if it was debated.
He said Mr Bercow had done "some very good things" during his time in the chair, but added: "One of the key roles of the Speaker is you must not speak on policy issues.
"He's broken that rule, he's broken his employment contract effectively with members of Parliament - he should resign, and if he doesn't, the vote of no confidence will go ahead."
He claimed there would have to be a vote to "resolve" his motion, although Early Day Motions tabled by MPs rarely make it as far as a Commons debate.
Mr Brake, a member of the House of Commons commission, said there had been a longstanding campaign by some Conservative MPs to get rid of Mr Bercow.
He said the Speaker's style "may feel like Marmite to some but in practice he is balanced".
On his blog, UKIP MP Douglas Carswell criticised Mr Bercow's "morally perverse" stance against President Trump but said he would not back an attempt to unseat him.
Mr Carswell pointed to Commons debates to come as the UK leaves the EU, adding: "Brexiteers should be exclusively focused on winning those fights - not picking others."
Before the recording of his Brexit comments emerged, Mr Bercow, who was a Conservative MP before becoming Speaker, was already facing calls for him to be replaced for voicing his opposition to US President Donald Trump addressing Parliament on his UK state visit.
Talking to Reading students, Mr Bercow said: "Personally, I voted to remain. I thought it was better to stay in the European Union than not."
He said this was "partly for economic reasons - being part of a big trade bloc - and partly because I think we're in a world of power blocs.
"I think for all the weaknesses and deficiencies of the European Union, it's better to be part of that big power bloc in the world than thinking you can act as effectively on your own."
He also said immigration was a good thing and expressed concern Labour had not done more to strike a "very clear, resonant Remain note".
When he was elected Speaker in 2009, Mr Bercow said he would serve for no more than nine years.
Responding to the row over his Brexit comments, his spokeswoman said he had voted in  June's referendum "along with millions of others".
"The record shows that he has rigorously facilitated the raising of concerns of those on both sides of this argument, as he does on every other issue," she said.
"The Speaker's impartiality is required on matters of debate before the House, and he has been scrupulous in ensuring that both sides of the argument are always heard."
Asked if the prime minister thought Mr Bercow was doing a good job, her spokesman said: "The Speaker is a matter for MPs, he's the elected chairman of the House of Commons, he's not a member of the government, he's not appointed by the government.
"He represents all the MPs in the House of Commons. It's up to them to decide whether they feel he's doing a good job or not."
A Conservative MP who is attempting to oust Speaker John Bercow says he should quit or face a vote of no confidence.