A Wake Forest graduate, Scarborough was used sparingly by the Washington Senators before World War II.
After winning 12 games for the Boston Red Sox in 1951, Scarborough was purchased by the New York Yankees in the 1952 midseason, as he went 5-1 during New York's successful pennant drive en route to the 1952 World Series.