The report warns that a failure to reintegrate returnees may lead to further radicalisation.
Somalia's Islamist al-Shabab militants are believed to be recruiting heavily in neighbouring north-eastern Kenya.
Kenya has seen a series of militant attacks with one at a university earlier this year killed 148 people.
Although the report does not mention where the returnees came from, Deputy Secretary General of the Supreme Council of Kenya Muslims (Supkem) Hassan Ole Nadu has confirmed to the BBC that they were fighting for al-Shabab.
The 36-page report was compiled by the IOM, Supkem and the Kenya's interior ministry.
Researchers found that although most adults joined the group voluntarily, nearly a third of children interviewed said they were forced to enlist.
The return of the youth, the report says, provides both "opportunities and threats for and to Kenyans".
"On the one hand, they present an opportunity to counter the radicalising ideologies and recruitment strategies of armed groups," the report says.
"On the other, the response of the security forces is widely perceived to be promoting further radicalisation as the returnees themselves believe that they are unsafe and have limited reintegration options."
Security in the country was identified by participants as the main obstacle to their successful reintegration in the country.
A total of 185 respondents were interviewed, representing nearly 30% of the estimated total of 684 returnees in the country. The majority were between 18 and 34 years old.
Nearly 700 recruits have returned to Kenya after quitting militant groups, a report by the International Organization for Migration (IOM) says.