 On the western shore is the Rocks district , the convicts ' side of the town , with two windmills on the ridge . The buildings were simple vernacular houses . Newly arrived convicts often found lodging in the district and in the 1790s a large number of the inhabitants were also Aborigines . To the left of The Rocks area is a long , low , military barracks ③ , built between 1792 and 1818 around Barracks Square / the Parade Ground – which is now Wynyard Park . It was from here that , in 1808 , the New South Wales Corps marched to arrest Macquarie 's predecessor William Bligh , an event later known as the Rum Rebellion . Heading east is St. Philip 's Church ④ – the earliest Christian church ( Church of England ) in Australia – erected in stone in 1810 on Church Hill – now Lang Park . In 1798 , the original wattle and daub church – on what is now the corner of Bligh and Hunter Streets – was burnt down , allegedly by disgruntled convicts in response to a decree by the second New South Wales Governor ( 1795 – 1800 ) , John Hunter , that all colony residents , including officers and convicts , attend Sunday services . The jail had earlier suffered a similar fate . 
