 His mother was an American painter, Lucy Ramberg (née Dodd), and his father a German archaeologist, Walter Ramberg.His father was killed while serving in World War I.

Ramberg and his mother moved from Italy to Munich for the remainder of World War I, and in 1920 they moved to his mother's family home in Portland, Oregon.He graduated from Lincoln High School in 1922.