He told the BBC the EU was "broken" and "undemocratic" and told the Sun many Labour councillors would "shock" Westminster with their referendum vote.
Veteran Labour MP Dennis Skinner has also said he will vote to leave the EU.
Labour's leadership is campaigning to stay in the EU and says Labour votes will be crucial in the referendum.
Senior party figures including Ed Miliband, Tom Watson and Yvette Cooper are attempting to mobilise Labour voters to back the Remain campaign later, with warnings of what they say is the potential economic impact of a vote to leave.
Among Labour voices suggesting the party must do more to get its supporters to back Remain are London Mayor Sadiq Khan and GMB leader Tim Roache.
This issue covers immigration and free movement within Europe.
But Mr Mann told BBC Radio 4's Today programme: "It's not that Labour's not getting its message across to Labour voters, it's that Labour voters are fundamentally disagreeing on this issue."
He said he had tried putting the case for and against in public meetings but had "found it impossible to argue the case for because the EU's fundamentally broken, it's undemocratic and even when you want to get changes - as David Cameron tried - you can't get them".
He said, on immigration, the EU's free movement of people did not allow the UK to plan for pressure on its public services.
The Bassetlaw MP, who has been critical of party leader Jeremy Corbyn, denied his stance was anything to do with "internal Labour Party politics" adding: "Jeremy Corbyn is far more in touch on this issue than [former leader] Ed Miliband - hence he's been more equivocal in some of the things he has said."
Tap here to enter your postcode.
He added: "Dennis Skinner has done an article in the Morning Star today - he's one of Jeremy's big supporters."
Writing in the Sun, Mr Mann said many Labour councillors would "shock" Westminster with their referendum vote and said a "people's revolution is under way" which was about "returning power to the people".
He wrote: "It is not sustainable to have 300,000 new people added to the population every year. It has created two kinds of people in this country: the people who gain from this and the people who lose out."
Bolsover MP Mr Skinner told the Morning Star he did not believe progressive reform of the EU could be achieved.
"My opposition from the very beginning has been on the lines that fighting capitalism state-by-state is hard enough. It's even harder when you're fighting it on the basis of eight states, 10 states and now 28.
"What [the EU] should be doing, if it wanted to convince people like me, is have a directive to get rid of zero-hours contracts across the whole of the EU. That's what I'd be looking for."
Sadiq Khan, the newly-elected mayor of London, said on Thursday that Labour had a "monumental responsibility" to ensure it mobilised supporters to vote in favour of continued EU membership.
"The worry that we have is, for understandable reasons because a lot of the media attention has been on so-called 'blue on blue' attacks - Boris Johnson versus David Cameron, Michael Gove versus George Osborne - Labour voters have been turned off," he said.
Labour MP John Mann has said he will vote to leave the EU and says Labour voters "fundamentally disagree" with the party's official position.