Andrew, and a younger brother, Matthew. His parents divorced when Olyphant was a teenager; both remarried. He is of English, German, Scottish, Dutch, Irish, and one-eighth Russian-Jewish ancestry. Olyphant is a descendant of the Vanderbilt family of New York. His paternal fourth great-grandfather was family patriarch Cornelius Vanderbilt; his third great-grandfather was William Henry Vanderbilt, who doubled the family's railroad fortune; his great-grandmother was socialite Emily Vanderbilt Sloane; and his great-uncle was music producer John Hammond. The surname Olyphant is of Scottish origin. Another paternal fourth great-grandfather, Dr. David Olyphant, was born in Scotland and served as director-general of the Southern hospitals during the American Revolutionary War. His third great-grandfather, David Olyphant, and great-great-grandfather, Robert Morrison Olyphant, were both prominent businessmen. Olyphant attended Modesto's Fred C. Beyer High School. Growing up, he was "embarrassed" by the idea of acting, but enjoyed art and drawing. He swam competitively throughout his childhood and was a finalist at the 1986 Nationals, in the 200m Individual Medley. He was then recruited to the University of Southern California by USC Trojans swimming coach Peter Daland. When Olyphant first visited the campus as part of a recruitment trip, he hoped to study architecture but was told it would be unmanageable with his training schedule. Instead, he opted to earn a Bachelor of Fine Arts. However, Olyphant left university one elective shy of a degree; he returned to finish the degree 30 years later, taking an online course during the COVID-19 pandemic. In 1990, he planned to finish his degree and apply for a master's degree in fine arts and half-heartedly considered a career in commercial art. While working as a swimming coach at Irvine Novaquatics, Olyphant decided to move to New York to explore other options. He initially performed stand-up comedy: "I'd dabbled [before] and then there was a six-month period where I did it with a certain commitment. Then I'd occasionally go back." Ultimately, he decided to become an actor. In his final year of college, he had taken an acting class as an elective at UC Irvine and found it "really enjoyable". He completed a two-year acting program at New York's William Esper Studio and began auditioning for roles. Career 1995–2003: Early supporting roles Olyphant's first paid acting job was in a 1995 WB television pilot based on 77 Sunset Strip. Phyllis Huffman cast him in the role but he did not have an