Closet where AC/Heater fan is located is very cold
There's a couple questions that have to be answered, before you'll know for sure if there's a problem or not.
First is if there's any combustion appliances in the closet (e.g. gas furnace)? If there are, they'll need to draw combustion air from somewhere.
Which leads to the second question, is there a duct to draw combustion air from the outside into the closet? Some furnaces draw air from the outside, but pull it directly into the combustion chamber. If this is the case in your situation, you don't have to worry about that. You'll only be concerned if there's a large open duct that leads to the attic, crawlspace, or outdoors.
If there are no combustion air ducts to the outside, then there's no problem with the air handler being in a conditioned closet. In fact, it's often recommended to install the HVAC system within the conditioned space.
In a lot of installations where the air handler is in the basement, you'll find a register within a few feet of the air handler.  This conditions the basement, bringing the HVAC into conditioned space.  Whereas when air handlers are in attics, they have to insulate the crap out of them. This is because the unit is in unconditioned space, so they use insulation to try and prevent the unconditioned air from interacting with the conditioned air.
tl;dr

If the closet is conditioned space, there may be no problem.
If the closet is not conditioned space, you'll want to seal up the ducts and insulate.

In either case, leaky ducts are not good. You'll want to seal them up with foil faced tape, and/or mastic.