The first prisoners came from Norfolk Island . They cut silos into the rock to store the colony 's grain supply . The initial prisoners were transferred to Cockatoo Island from Norfolk Island , and were employed constructing rock-cut silos for storing the colony 's grain supply .