he had come out to stay with them for a couple of weeks , but whether it was because of the row he had had with harry while he was there or just that he could not bear to see the deterioration in his brother , he never repeated the visit .