Plum Island is an island in the Town of Southold in Suffolk County, New York in the United States.The Island is situated in Gardiners Bay, east of Orient Point, off the eastern end of the North Fork coast of Long Island. It is about 3 miles (4.8 km) long and 1-mile (1.6 km) wide at its widest point.he northern portion of Plum Island is a recessional moraine deposit, and is part of the Harbor Hill-Roanoke Point-Fishers Island-Charlestown Moraine, thus is one of the Outer Lands. Boulders in the moraine can be seen in the eroding northern slope of the island.