the more illustrious of colum 's tacksmen and tenants were housed in the castle proper , while the poorer men-at-arms and cottars set up camp on a fallow field below the stream that fed the castle 's loch .