In 1842 he settled in Hong Kong and became essential to the colonial administration in dealing with the local Chinese in the earliest years of Hong Kong as a British colony.
Soon after the battle, Caldwell resigned from the government due to low pay and purchased a merchant steamer, The Eaglet, which ran coastal trade and set up convoys escorting junks along the South China coast.