Nevertheless, when the Nazi regime gained power decades after Hertz's death, his portrait was removed by them from its prominent position of honor in Hamburg's City Hall (Rathaus) because of his partly Jewish ethnic ancestry.
Hertz's widow and daughters left Germany in the 1930s and went to England.