Brown was born in Holtville, California, one of four children of George Edward Brown Sr. and Bird Alma Kilgore.
Shortly after the Japanese attack at Pearl Harbor, Japanese Americans were sent to internment camps, an action that offended Brown so much that he helped organize protests in Los Angeles in 1942.