The success of generative models for images, text, and videos has sparked interest for their application for the direct generation of three-dimensional candidate structures for molecules and materials. Despite some success, the field is still in its infancy. Most early approaches adopt a global representation of the full molecule or unit cell, requiring a large amount of training data. Only recently, models based on local environments have begun to appear. These are however limited to molecular, non-periodic structures and it is unclear how a step-wise atom placement approaches can be generalized to periodic structures.