Brown was born in Glasgow, Scotland, where his father was a Clydeside engineer.
In 1947 he married Mary Carmichael, daughter of ILP and then Labour MP Jimmy Carmichael and sister of future Labour MP and peer Neil Carmichael, and he served as a councillor on the Glasgow Corporation from 1954.