he knew the duke of perth 's court would not be unduly severe on his men , at least this year , so leaving a small party of the balquhidder men to wait for the stragglers , he led the remainder homewards .