1. A semiconductor memory device comprising: a semiconductor substrate; a stacked body provided on the semiconductor substrate and having a plurality of insulator layers and a plurality of conductive layers alternately stacked; a semiconductor layer provided inside a through-hole formed so as to pass through the stacked body and extending in a stacking direction of the plurality of insulator layers and the plurality of conductive layers; and a charge trap layer provided between each of the conductive layers and the semiconductor layer, a plurality of memory cells being connected serially in the stacking direction, each of the memory cells including each of the conductive layers, the semiconductor layer and the charge trap layer between each of the conductive layers and the semiconductor layer, the semiconductor layer in the plurality of memory cells having an upper part and a lower part narrower than the upper part, a lower layer, in the plurality of conductive layers, facing the lower part of the semiconductor layer being thinner than an upper layer, in the plurality of conductive layers, facing the upper part of the semiconductor layer.