1. A semiconductor memory device comprising: a semiconductor layer functioning as a rectifying device; a variable resistance layer provided above or below the semiconductor layer, and reversibly changing its resistance; a sidewall layer physically contacting a sidewall of the semiconductor layer and the variable resistance layer; and a buried layer embedded in the sidewall layer, and being made of material different from that of the sidewall layer, wherein the sidewall layer comprises: a sidewall semiconductor layer contacting a sidewall of the semiconductor layer, the sidewall semiconductor layer being different from the semiconductor layer; and a sidewall insulating layer provided on a side surface of the sidewall semiconductor layer via the buried layer, wherein the buried layer is continuously formed to cover the semiconductor layer and the variable resistance layer.