What can I do to reduce cold infiltration by a direct-vent gas fire place?
It is very important to ensure all infiltration paths a sealed off and adjacent walls and floor are properly insulated. Even a hairline crack in the doors can cause enough infiltration to lower temperatures in the area, though no actual moving air can be detected.

Even so, the fact that outside air can freely circulate in the combustion chamber will also be a heat sink, lowering temperatures in the area. More sophisticated devices similar to this utilize automatic dampers that interlock with the fuel supply. These are located on the exhaust flue, and sometimes on the combustion air inlet as well. I don't know if retrofitting such devices is possible on typical residential fireplaces. Such a project must be done very carefully to ensure complete safety. Failure of the system can result in carbon monoxide poisoning, so multiple fail safe measures are called for. Such measures are beyond the abilities of nearly all DIY practitioners. Only fully qualified professionals should perform this sort of work.

Even more so, though flue dampers prevent cold air from freely circulating, they are typically thin sheets of metal with poorly sealed edges, so the fire box will still be fairly cold, albeit much less so.