1. A semiconductor memory device comprising: a semiconductor substrate; a stacked body provided on the semiconductor substrate and having a plurality of insulator layers and a plurality of conductive layers alternately stacked; a semiconductor layer provided inside a through-hole formed so as to pass through the stacked body and extending in a stacking direction of the insulator layers and the conductive layers; and a charge trap layer provided between the conductive layer and the semiconductor layer, a lower part in the semiconductor layer being narrower than an upper part therein, and at least the lowermost layer in the conductive layers being thinner than the uppermost layer therein.