A major early influence on Brown's bass playing was Jimmy Blanton, the bassist in the Duke Ellington band.
From 1974 to 1982, Brown performed and recorded a series of albums with guitarist Laurindo Almeida, saxophonist and flautist Bud Shank, and drummer Shelly Manne (replaced by Jeff Hamilton after 1977) under the name The L.A. Four.