Vyrubova was born in Oranienbaum, the daughter of Aleksandr Taneyev, Chief Steward to His Majesty's Chancellery and a noted composer.
She was a childhood playmate of Felix Yussupov, the man who spearheaded the murder of Grigori Rasputin.