 Brown was born in Holtville, California, one of four children of George Edward Brown and Bird Alma Kilgore.Brown graduated from Holtville Union High School in 1935 and attended Central Junior College (now Imperial Valley College) in 1938.He then entered the University of California, Los Angeles (UCLA), where he became head of the UCLA Student Housing Association and helped found the University Housing Cooperative Association (UHCA), a student housing cooperative, in 1938.