to the irritation and annoyance of the original owners , who were seriously outnumbered , the sergeant carefully backed away from the confrontation and waved the remaining carts to continue their journey .