1. A method of manufacturing semiconductor memory device, comprising: forming a first wiring layer above a semiconductor substrate; forming a first memory cell layer above said first wiring layer, said first memory cell layer being arranged to configure first memory cells therein; forming a plurality of first trenches in said first wiring layer and said first memory cell layer, said first trenches extending in a first direction, thereby forming first wirings extending in the first direction and separating said first memory cell layer by said first trenches; burying a first interlayer film in said first trenches to form a first stacked body; forming a second wiring layer above said first stacked body; forming a plurality of second trenches in said first stacked body with said second wiring layer formed thereabove, said second trenches extending in a second direction intersecting the first direction and reaching an upper surface of said first interlayer film in depth, thereby forming second wirings extending in the second direction; removing said first interlayer film isotropically from a portion exposed in said second trenches toward the interior; and digging said second trenches down to an upper surface of said first wirings after removing said first interlayer film isotropically, thereby forming said first memory cells in the shape of pillars separated by said first and second trenches at the intersections of said first and second wirings.