Addresses were searched in Livingston, Bathgate, Blackburn and Salford, with a reception centre set up to support potential victims.
Police Scotland said 25 people were interviewed as part of the operation.
The men are due to appear at Livingston Sheriff Court on Monday. Two of them are aged 28 and the others 35 and 26.
Police Scotland was supported in the operation - which took place on Friday - by the National Crime Agency, Greater Manchester Police and West Lothian Council.
The men were detained at addresses in Livingston, Bathgate and Salford in Greater Manchester and have been charged in connection with human trafficking offences, involvement in serious and organised crime, and a sexual offence.
Six further addresses were searched in Livingston, Bathgate and Blackburn.
Det Ch Insp Stephen Healy said: "In total, twenty-five people have been interviewed as part of this operation and enquiries are ongoing to establish the circumstances surrounding their travel to and from Scotland.
"In the meantime, they are being offered support and assistance in terms of health and welfare, housing and victim support.
"This operation is the result of a complex and lengthy investigation into serious and organised crime involving foreign nationals centred on West Lothian."
Ch Insp Barry Blair, West Lothian local area commander for Police Scotland said: "The activity under Operation Heathyard has been centred on the Latvian communities in West Lothian and we will continue to support all those affected going forward.
"We will have additional officers on patrol in these neighbourhoods over the weekend to provide reassurance and to speak to anyone who may have any further information or who needs any more advice and assistance.
"I would urge anyone with information or concerns about human trafficking or exploitation in their community to speak to police on 101. We will do all we can with our partners to keep people safe."
West Lothian executive councillor for services for the community, George Paul, added: "Along with other public sector partners, West Lothian Council will provide social care support to the victims."
Four men have been arrested and charged after an operation aimed at tackling human trafficking and serious organised crime in West Lothian.