higgins tells me that no man would dare go out alone in uniform , for fear of these mobs , and that even when in greater numbers , harassment from the public soon drove them back to their quarters , save when compelled by duty to persist .