During the later years of his ministry his relationship with the British colonial government was tense; Kagwa believed Buganda's autonomy was being repeatedly encroached upon.
He had 23 children, including Michael Kawalya Kagwa (who served as Buganda's Katikiro from 1945 to 1950)[REF] His grandchildren include Apollo Kironde, Uganda's first Ambassador to the United Nations.