What is the easiest way for a new private pilot to visit Manhattan?
All of NYC's airspace is crowded, so there's no escaping that! :) The controllers are very friendly and accommodating though, so don't be afraid. If they're curt, it's just to keep everyone safe. As for airports, there are 4 GA airport options you're most likely gonna consider:


KLDJ in Linden, NJ. It's a 5m cab ride to the train station into the city. Uncontrolled field, tons of news helicopters based here. It's pretty cheap (around $20/night if I recall) and fairly convenient, but the uncontrolled field is under an 800' Class B shelf at Newark and could be stressful. Expect very bare bones facilities - you can get fuel, but there's no terminal building to speak of.
KTEB in Teterboro, NJ is the GA airport for most jets that come into the city. It's the closest airport to midtown, and you'll pay a very pretty penny here, tie-downs are at least \$50/night, and fuel will be around \$9/gallon. This is a hectic option, but it has a tower and is close to the city, so for a quick overnight stop without fuel it may be the best option. Expect to be the only piston on the ramp.
KFRG in Farmingdale, NY (Long Island). Very busy GA airport abuzz with training. Pretty cheap (~$15/night tiedowns), but did I mention that it's busy? You may burn up 1h of hobbes time on the ground waiting to take off if you depart during peak times. You can catch a cab to LIRR which will get you into the city.
KHPN, in White Plains, NY. Mix of GA, training, and a couple commercial carriers like JetBlue. Prety cheap ($~20/night tiedowns) and a short hop away from Metro-North which will have you in the city in about 30 minutes. Controlled field, at 2500' below the Class B shelf. Fairly busy, but certainly managable with the top notch controllers working NY's airspace.


Whichever airport you choose to go to, absolutely get flight following or file IFR. All of the towers in the area expect you to be handed off from NY Approach and will not be happy if you call them directly out of the blue.