Longford in Hillingdon could be demolished if a third runway is built at neighbouring Heathrow Airport.
A hotel is providing accommodation in houses it has bought for asylum seekers, including those from the so-called "Jungle" in Calais.
The Home Office said it has tried to address residents' concerns.
It has said decisions about the use of hotel accommodation, including which premises are used, are a matter for the contractors who bear the cost.
It would not say how many asylum seekers were being sent to Longford, but said rooms were used for one or two days before asylum seekers were dispersed into temporary accommodation.
BBC London filmed two coaches and three van loads arriving in one day carrying people from Iran, Sudan, Eritrea and Syria.
Many of those on board said they had come from the so-called "Jungle".
Three days later, another two coaches were witnessed arriving.
Ray, 85, has lived in Longford for 50 years. He said asylum seekers were accommodated in the houses either side of his.
"They meet and they have conversations in the middle of my front garden," he said.
"We don't know where they come from, we don't know what they are and we are living next door to them, albeit they might be very nice people."
The hotel, Heathrow Lodge, also rents rooms to travellers using the airport.
Booking websites confirm a room can cost Â£30 a night per person, less per person if sharing.
Neither the Home Office nor the hotel would say how much is paid per asylum seeker.
BBC London booked a room and met eight men all staying in one house.
They were asylum seekers from Sudan and Eritrea and said a few days earlier they had been sleeping rough in the camp in Calais.
Mogdad from Sudan said he had paid a people trafficker 500 euros (Â£370) in Calais to get to the UK.
"I have no choice," he said. "I have to get away from death."
Villager Ray said: "The coaches turn up  - about 20, 30 people get off, and they're transferred to all the houses that are around the village."
Longford resident Leon, said: "You know, (with) what's going on around the world at the moment, they've got a right to move from dangerous areas, but it's just saturated where I live, with immigrants.
"I chose to live here with the aeroplanes, but I didn't choose to live here with the three to four coaches a day moving in and out, and especially at night-time as well.
The villagers insist they are not being racist.
Rana Saif, who runs one of the local pubs was born in Pakistan and moving to Longford from Sweden three years ago.
"No racists here," he said.
He claimed the numbers of asylum seekers staying in the village had hit his takings.
"No-one comes here, when they are standing 20, 80 people outside on the road, on the walls.
"I am going to move from here, as soon as.
"It was good before the immigrants came."
While the future of the village may be bound by the government's decision over whether to build a new runway at Heathrow, residents say they want something done.
The Home Office said it had been made clear to contractors that the use of hotels was only ever acceptable as a short-term contingency measure.
Hillingdon Council said it received three complaints earlier this year, adding it had looked into claims rooms were being overcrowded and had put in additional licensing arrangements for the hotel.
The hotel manager did not wish to be interviewed or to give a statement.
However, in a recent meeting with some villagers he said he was sorry people were unhappy. He wanted to make it better, and in future no more than 40 asylum seekers would be staying at any one time.
Residents of a village near Heathrow Airport say they are being "overwhelmed" by people seeking asylum in the UK.