Fifa delegates are voting on their new president, but the incumbent Mr Blatter has refused to withdraw from the contest, despite the arrest of senior colleagues on major corruption charges.
The PM said Fifa could only rebuild its credibility if Mr Blatter left.
It comes as the UK's Serious Fraud Office (SFO) said it was looking at material relating to the allegations.
Seven senior Fifa officials were arrested at a hotel in Switzerland on Wednesday.
The US Justice Department (DOJ) has charged 14 people with racketeering, wire fraud and money laundering conspiracy.
Mr Cameron's comments followed Mr Blatter's call for unity at Fifa as delegates prepare to vote on his presidency.
Mr Blatter, who is expected to win a fifth term, said: "I am appealing to unity and team spirit so we can move forward together."
But speaking in Berlin, Mr Cameron said: "You cannot have accusations of corruption at this level and on this scale in this organisation and pretend that the person currently leading it is the right person to take it forward.
"That cannot be the case. Frankly, what we've seen is the ugly side of the beautiful game and he should go.
"And the sooner that happens the better - the faster that organisation can start to rebuild its credibility, which is going to be so important because so many people around the world want to see this game properly managed, properly looked after, so we can all enjoy the World Cups of the future."
Mr Cameron, who is holding EU reform talks with German Chancellor Angela Merkel, said it was "unthinkable" that Mr Blatter should stay.
He also said he would be "guided" by the Football Association (FA) chairman, Greg Dyke, over the question of whether the bids for the 2018 and 2022 World Cups should be opened.
Mr Dyke suggested earlier that he would not rule out a boycott of the major football tournaments if Mr Blatter remains at the helm of Fifa.
The Union of European Football Associations (Uefa), Europe's governing body, may consider boycotting future World Cups if Mr Blatter wins Friday's vote, according to Scottish FA chief executive Stewart Regan.
Meanwhile, the SFO said it was actively assessing "material in its possession" relating to Fifa corruption allegations.
It said it had "made plain that it stands ready to assist ongoing international criminal investigations".
It said its probe had been ongoing for months, before Wednesday's indictments.
Despite a global media and sporting storm, Fifa president Sepp Blatter is seeking re-election for a fifth term, with the only challenge coming from Jordan's Prince Ali bin al-Hussein.
The vote is taking place at the Fifa World Congress meeting in Zurich, Switzerland.
The SFO said it was not willing to comment further on the nature of material in its possession or current activity in this area.
It added that it had not launched a formal criminal investigation.  But it said there were two ways in which it could get involved - if the SFO opened a formal criminal investigation itself or provided "mutual legal assistance" to overseas law enforcement.
It also says it has made plain to both US and Swiss authorities that it stands ready to assist.
"There are currently international criminal investigations into allegations of corruption at Fifa. As such it would be inappropriate for the SFO to comment on the matter at this time," it said.
In November the SFO asked for anyone with information regarding alleged corruption in the bidding for the 2018 and 2022 World Cups to come forward.
However, it will not now say if its current actions relate to either of the above tournaments.
David Cameron has said Fifa president Sepp Blatter must resign, adding "the sooner that happens the better".