Stone was dug up on the island to build the prison as well as other buildings in Sydney , including the seawall for Circular Quay . Later , quarrying on the island provided stone for a group of penal buildings on the island and for construction projects around Sydney , including the seawall for Circular Quay .