It also warns that Britain may have to allow EU-registered boats to fish in UK waters as part of an overall deal.
Fishing regions around the UK voted heavily in favour of leaving the EU during the referendum campaign.
The Lords review says these communities are at risk of being marginalised in the wider Brexit negotiations.
The EU's Common Fisheries Policy (CFP), with its quotas and principle of equal access to commercial fishing grounds for boats from all member states, has often been characterised by the industry as a disaster for Britain.
This dislike helped mobilise many in the industry to campaign for a leave vote in the referendum last June.
Many in the fishing community argue that Brexit now offers the industry the chance to regain control over UK waters and become a leading fish-exporting nation, like Norway.
However, the House of Lords European Union Committee has released a report that looks at the risks and opportunities for the UK industry.
Since UK fishing only produces a half of one percent of GDP and employs just 12,000 fishers, the Lords say that industry might be a low priority for the government but it "must not be marginalised in the wider Brexit negotiations".
What complicates the picture is the fact the most commercial fish stocks are in waters that are shared between the UK and other EU coastal states. The vast majority of UK fish are exported, mainly to the EU while a significant proportion of the fish that British consumers eat is imported, often from EU states.
"A successful industry," the report says, "therefore needs continued market access."
However, that access may come at a price.
"Brexit will involve many trade-offs," said Lord Teverson who chairs the Lords EU Energy and Environment sub-committee.
"It may very well be that EU member states demand more access to UK waters than some fishers would want in return for our continued rights to sell fish to the European market with zero tariffs."
The report also points to the fact that many elements of the CFP should be retained post-Brexit and the UK should ensure that total allowable catches and quotas should continue to be based on scientific advice. The UK will also continue to be under international obligations to co-operate with neighbouring states.
One area of concern, according to the report, may well be that if Britain leaves the EU, the CFP framework which treats the UK as one entity will fall away "raising the potential of four different fisheries management regimes" in England, Wales, Northern Ireland and Scotland.
However, many in the fishing industry still argue that Brexit will bring more opportunities than threats.
Just this week, the EU announced agreement on new quotas under the CFP - many analysts say that it marked an advance for the UK with significant increases in allowable catches for plaice, haddock and prawns.
Responding to the new quotas, representatives from the industry said they believed this augurs well for the future of British fishing outside the EU.
"With Brexit now looming, fishermen can look to the future with real optimism as we are on the cusp of an exciting new era as a coastal state with full control of our 200-mile exclusive economic zone," Bertie Armstrong, chief executive of the Scottish Fishermen's Federation said in a statement.
"This will give us the opportunity for fairer shares in catching opportunity and better fit-for-purpose sustainable fisheries management, which will benefit our coastal communities."
In response to the report, a government spokesperson said: "As we enter the EU negotiations, the prime minister has been clear we want to ensure British companies have the maximum freedom to trade with and operate in the single market."
"At the same time, leaving the EU is a real opportunity to review fisheries management in order to ensure fair access to quota, sustainable stocks and a healthy marine environment."
Follow Matt on Twitter and on Facebook
The UK fishing industry will need continued access to EU markets if it is to thrive after Brexit, a House of Lords report has warned.