Tolsdorf was born on 3 November 1909 in the family estate in Lehnarten in the Province of East Prussia, a state of the German Empire.
Tolsdorff was five years old in 1914 when his mother briefly evacuated the estate following the invasion of East Prussia by the Russian First Army, led by Paul von Rennenkampf.