The vessel - believed to be "The Harvester", from Milford Haven - got into trouble on rocks at St David's Head at about 14:30 BST on Thursday.
The rescued man was taken to hospital but was pronounced dead, while the search for a second man was scaled back on Friday afternoon.
Wreckage from the boat has been found over a four mile (6.4km) area.
Dyfed-Powys Police sergeant Gareth Walters called it a "tragic incident".
Alex Smith, National Maritime Operations controller for the Coastguard said: "Despite an intensive search including the helicopter, RNLI lifeboats and Coastguard rescue teams we have been unable to locate the remaining missing crew."
Five lifeboats, a coastguard helicopter using thermal image cameras and cliff teams have been involved in the search.
Speaking to BBC Radio Wales earlier on Friday, Jim Phillips, from the RNLI in St Davids, said a member of the public on the coastal path saw the boat smashed on a rock with two people in the water at about 14:40 BST.
The vessel was about a mile (1.6km) out to sea near Abereiddi.
Mr Phillips said by the time rescuers got to the scene it had sunk and all that was left on the water's surface was debris.
Dyfed-Powys Police appealed for any witnesses to come forward.
A fisherman has died and another is still missing after a boat sank in Pembrokeshire, police have confirmed.