The UNHCR said forced returns had "continued unabated" despite an agreement earlier this month.
Under the deal, any returns would be voluntary and only "when conditions were conducive".
Cameroon has rejected the accusation and said people returned willingly.
According to the UNHCR, more than 2,600 refugees have been forcibly returned to Nigeria from Cameroon this year.
Many are unable to go back to their villages in Borno state for security reasons and have ended up in camps for displaced people.
In some cases, the UNHCR said, people had been returned "without allowing them time to collect their belongings".
UNHCR spokesman Babar Baloch spoke of "chaos" in the returns process and said "some women were forced to leave their young children behind in Cameroon, including a child less than three years old".
Many of the returnees are now settled in the Banki camp for internally displaced people.
UNHCR staff also recorded about 17 people who claimed to be Cameroonian nationals, who it said had been deported by mistake to Banki.
It is common in the region to find people who have no documentary proof of their nationality.
Cameroonian Communications Minister Issa Tchiroma told the BBC's Focus on Africa programme: "I strongly deny this accusation" of forced returns.
He said the Cameroonian army had been working "hand-in-hand" with the Nigerian army against Boko Haram and any civilians who had returned to Nigeria had done so of their own accord.
"This repatriation has taken place willingly," he said.
The Cameroonian authorities have previously said Boko Haram militants have been entering the country disguised as refugees.
Militants have carried out a number of attacks in northern Cameroon in recent years, often using suicide bombers.
The UNHCR said forced return constitutes a serious violation of the 1951 Refugee Convention and the 1969 OAU Convention, both of which Cameroon has ratified.
It called on Cameroon to honour its obligations under the conventions and continue keeping its borders open so as to allow access to territory and asylum procedures for people fleeing the Islamist insurgency.
The UN refugee agency has criticised Cameroon for the forced return of hundreds of refugees to north-east Nigeria after they had fled from the Islamist Boko Haram insurgency.