though he usually tried not to cause such an uproar in thehighlands that robert would be called upon to intervene , for he knew it would mean trouble for his brother when he refused to oust the renegades and broken men from creag dhu , niall 's patience was at an end .