 Further along the ridge to the east is Fort Phillip , flying the Union Jack , on Windmill ( later Observatory ) Hill where the Sydney Observatory is now located . It was the highest point above the colony , affording commanding views of the Harbour approaches from east and west . On the waterfront below Fort Phillip is the yellow , four @-@ storey , Commissariat Stores , ⑤ , constructed by convicts for Macquarie in 1810 and 1812 . One of the largest buildings constructed in the colony at the time , it is now the site of the Museum of Contemporary Art . The foreshore buildings on the extreme right are the warehouse and " Wharf House " residence of merchant , Robert Campbell ⑥ who was to become one of the colony 's biggest landholders . This is now the site of the Sydney Harbour Bridge pylons and is just to the left of Dawes Point . Three British sailing ships , flying either the Red Ensign of the Merchant Navy or ( more likely ) the White Ensign of the Royal Navy , are anchored in the Cove along with four sailboats and five canoes . 
