The issue is being made worse by a lack of "fair and equal dispersal", with some areas taking hundreds of people and some taking none, its MPs said.
They condemned an "appalling" episode where asylum seekers were told to wear wristbands, and an instance where their doors were all painted one colour.
The Home Office said it was committed to providing safe and secure housing.
"The dispersal system appears unfair, with whole swathes of the country never receiving a single asylum seeker," said committee chairman Keith Vaz.
"The majority are being moved into low-cost housing in urban areas such as Glasgow, Stoke, Cardiff and of course Middlesbrough, where the ratio is 1 asylum seeker per 137 people.
"However, on the data we have received, local authorities in areas such as Maidenhead, Lincoln and Warwick have housed none."
Other areas listed in the report as having no-one in receipt of aid given to asylum seekers - known as Section 95 support - in the final quarter of 2015 included Cambridge, Cornwall, Midlothian and York.
Local authorities with few asylum seekers, or none, should be "actively encouraged by ministers to volunteer" to take some, the MPs said.
How many asylum seekers live in your area?
At present, 200 local authorities are part of a scheme that provides housing for asylum seekers.
The three private firms that operate the programme - known as Compass - the Home Office, and the home affairs committee all agree that getting more councils on board is the key to easing the chronic accommodation shortage. It may well become a crisis if the number of asylum applicants continues to increase.
At 38,000 the rate is at its highest since 2004.  Some councils would take part if the contractors could find enough suitable and affordable properties; in other areas, it may be the fear of a voter backlash which is putting off local politicians from agreeing to participate.
With the government committed to resettling 20,000 Syrians by 2020, as well as accommodating asylum seekers, ministers will have to find a solution - or face the prospect of housing more of them in two-star and three-star hotels.
The committee said its findings "leave major questions to be answered" about the running of Compass - the name of the contract for providing asylum support services.
The main contractors involved are G4S, Clearsprings and Serco, and the committee said delivery "has been mostly unsatisfactory to date".
"G4S, Serco and Clearsprings receive millions of pounds of public money to house asylum seekers, and revelations in recent months have been alarming," Mr Vaz said.
"The compulsory wearing of wrist bands and the infamous red doors demonstrate an unacceptable attitude towards vulnerable people.
"The Home Office has failed to provide proper oversight and inspection, and must do better."
The Home Office said it was working with areas that had not taken part so far in asylum dispersal.
A spokesman said: "The Home Office has worked closely with our providers to improve property standards over the lifetime of the Compass contract.
"Where a contractor is found to be falling short of these standards, we work with them to ensure issues are quickly addressed. When they are not, we can and do impose sanctions.
"All suppliers are meeting the contractual requirements to provide safe, habitable, fit for purpose and correctly equipped accommodation in compliance with the Housing Act 2004 and the Decent Homes Standard."
There is an "impending shortage" of housing for asylum seekers in the UK, the home affairs committee has warned.