The Continental Army , the army that had fought in favor of the thirteen colonies led by George Washington , had to leave Manhattan after the Battle of Fort Washington on 16 November 1776 . The Continental Army was forced to abandon Manhattan after the disastrous Battle of Fort Washington on November 16 , 1776 .