Labour MP Yvette Cooper said it was not "appropriate" or "statesmanlike" for him to use such language about a "complex and sensitive" issue.
Shadow minister Kate Green said it was "offensive, hurtful... and divisive".
Mr Cameron said Labour leader Jeremy Corbyn had told the migrants at the camp "they could all come to Britain".
At the weekend, the Labour leader visited the Grande-Synthe Camp, near Dunkirk, and The Jungle in Calais, where more than 7,000 people are sleeping rough, describing conditions there as "disgraceful".
Jeremy Corbyn had a good go at David Cameron over the Google tax deal, using a crowd-sourced question from Jeff (or possibly Geoff), who wanted to know whether he could join a scheme allowing him to pay the same rate of tax as the internet giant.
But some commentators felt the Labour leader lost focus when he switched to other subjects for his final two questions.
"We've had no answer on Google, and no answers on Jeff," said Mr Corbyn, before turning to what he calls the "bedroom tax" and a High Court ruling that it is "discriminatory". For his final question, he called on the PM to launch an inquiry into arms export licences to Saudi Arabia amid UN reports of coalition air strikes on civilians in Yemen.
The prime minister responded to that last point by saying the UK had some of the strictest rules for arms exports in the world. He added that Britain was "not part of Saudi-led operations" in Yemen.
But it was one of those weeks when reaction to what was said at PMQs overshadowed anything that was said during the session, as Labour seized on Mr Cameron's "bunch of migrants" comment.
Mr Cameron was coming to the end of a clash with Jeremy Corbyn about Google's tax deal with the government, when he decided, as he often does, to broaden out his attack to other issues.
The Labour leader had just accused him of failing to stand up to Google.
Mr Cameron replied: "The shadow chancellor's pointing - the idea that those two right honourable gentlemen would stand up to anyone in this regard is laughable.
"Look at the record over the last week - they met with the unions and they gave them flying pickets.
"They met with the Argentinians, they gave them the Falkland Islands.
"They met with a bunch of migrants in Calais, they said they could all come to Britain.
"The only people they never stand up for are the British people and hard-working taxpayers."
Several Labour MPs objected to Mr Cameron's "bunch of migrants" comments on Twitter, describing them as "shameful" and "inflammatory" and the PM as "odious".
A Labour spokesman said Mr Cameron's use of the phrase "demonstrates an attitude that is wholly unacceptable to a humanitarian crisis on our doorstep".
Yvette Cooper, who is chairing Labour's refugee task force, said Mr Cameron should use "much more statesmanship-like" language on such a "complex and sensitive" issue, particularly given ongoing commemorations of Holocaust victims.
She got to her feet immediately after Prime Minister's Questions to raise a point of order
She asked Commons Speaker John Bercow to ask Mr Cameron to "withdraw" the comment.
Mr Bercow told Ms Cooper: "You speak with enormous experience in this House and I respect what you say.
"I completely identify and empathise with your observations about Holocaust Memorial Day, which you and I on other occasions have marked at events together - so I take what you say extremely seriously."
He ruled that it was not "disorderly" or "un-Parliamentary" but added "people will make their own assessments of this matter".
The prime minister's spokesman said the government had spent Â£1.2bn to help hundreds of thousands of refugees in camps in the region and he said Labour's approach would "open the door and provide an incentive for more to come to Calais".
Asked if he considered the phrase "bunch of migrants" to be pejorative the spokesman said "the PM thinks that the key thing here is getting the policy right".
Mr Cameron was accused by some opponents of using the "bunch of migrants" phrase to distract attention from criticism of the Google agreement.
But this was dismissed as "silly and playing cheap politics" by small business minister Anna Soubry
She told BBC Radio 4's The World at One Mr Cameron had been using the "language of ordinary people", adding "we all use slang" in the heat of the moment.
David Cameron insisted he has done more than any other prime minister to crack down on aggressive tax avoidance and blamed Labour for failures in collecting taxes from large multinationals.
Mr Corbyn claimed Google was paying an effective tax rate of 3% - and accused the government of sending out mixed messages, with Chancellor George Osborne initially describing the deal as a "major success" before No 10 rowed back, calling it a "step forward", while, he said, London mayor Boris Johnson labelled it "derisory".
Mr Cameron then went on to explain the measures the Government has put in place to tackle tax avoidance and evasion before claiming the tax rate for Google under Labour was "0%".
Mr Corbyn also pointed to a series of meetings Google had had with Government ministers.
"Millions of people are this week filling in their tax returns to get them in by the 31st," he said.
"They have to send the form back, they do not get the option of 25 meetings with 17 ministers to decide what their rate of tax is.
"Many people going to their HMRC offices or returning them online this week will say this - 'Why is there one rule for big multinational companies and another for ordinary, small businesses and self-employed workers?"'
Mr Cameron brushed off the criticism by launching into an attack on all the things Labour did not stand up to, including migrants.
Firstly: George Osborne risked sounded far too content when he hailed the deal as a "victory" and a "major success". Government spokesmen were reluctant to repeat his verdict.
Secondly: Labour reacted quickly and managed to get a hearing. Their message sounded louder than their internal disputes, for a change.
Thirdly: voters care. The perception international firms get a better deal than ordinary people is toxic.
HMRC collects tax, not ministers, and the government says it has acted and got results where Labour did not.
But few politicians ever caught the mood of a nation declaring themselves happy with a big business's tax return.
Mr Cameron began Prime Minister's Questions by announcing that a "striking" memorial to the victims of Nazi atrocities is to be erected beside Parliament, in Victoria Tower Gardens, where a number of monuments already stand.
Downing Street said an international design competition will be launched in the coming weeks and the memorial built by the end of 2017.
"It will stand beside Parliament as a permanent statement of our values as a nation and will be something for our children to visit for generations to come," Mr Cameron said.
"It is right that our whole country should stand together to remember the darkest hour of humanity."
The SNP again challenged the prime minister to help those women born in the 1950s, affected by pension equalisation, with the party's leader at Westminster using both of his questions to press the point.
He called on David Cameron to respect a Parliamentary decision to immediately introduce transitional arrangements for those women negatively impacted (the vote was non-binding and does not compel ministers to act).
Mr Cameron did not address the point directly, but claimed the introduction of a single tier pension at Â£155 a week would end discrimination because "so many women retiring will get so much more in their pension".
Liberal Democrat leader Tim Farron is guaranteed one question at PMQs every month and he used his latest opportunity to urge the prime minister to do more to alleviate the refugee crisis facing Europe.
"The only way to challenge a crisis of that magnitude is by starting working with our European colleagues at the heart of a united Europe," he said.
He urged Mr Cameron to "welcome in and provide a home" to 3,000 unaccompanied children as recommended by campaigners.
In response, Mr Cameron said no country in Europe had been "more generous" in supporting refugee camps in the Middle East but the UK would not be taking part in the EU's refugee relocation and resettlement schemes as it was not in its national interest.
David Cameron is facing Labour calls to apologise for calling refugees at a camp in Calais "a bunch of migrants" at Prime Minister's Questions.