DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraft
Sam Earle Filippos Kokkinos Yuhe Nie Julian Togelius Roberta Raileanu































  New York University
  Broklyn
  USA




 Meta
 London
 UK
 


 New York University
 Brooklyn
 USA
 


 New York University
 Brooklyn
 USA
 


 Meta
 London
 UK





























































Procedural Content Generation (PCG) algorithms enable the automatic generation of complex and diverse artifacts. However, they don't provide high-level control over the generated content and typically require domain expertise. In contrast, text-to-3D methods allow users to specify desired characteristics in natural language, offering a high amount of flexibility and expressivity. But unlike PCG, such approaches cannot guarantee functionality, which is crucial for certain applications like game design. 
In this paper, we present a method for generating functional 3D artifacts from free-form text prompts in the open-world game Minecraft. Our method, , trains quantized Neural Radiance Fields (NeRFs) to represent artifacts that, when viewed in-game, match given text descriptions.
We find that  produces more aligned in-game artifacts than a baseline that post-processes the output of an unconstrained NeRF. 
Thanks to the quantized representation of the environment, functional constraints can be integrated using specialized loss terms. We show how this can be leveraged to generate 3D structures that match a target distribution or obey certain adjacency rules over the block types. 
 inherits a high degree of expressivity and controllability from the NeRF, while still being able to incorporate functional constraints through domain-specific objectives.







<ccs2012>
   <concept>
       <concept_id>10010405.10010469.10010474</concept_id>
       <concept_desc>Applied computing Media arts</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010257.10010293.10010294</concept_id>
       <concept_desc>Computing methodologies Neural networks</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Applied computing Media arts
[300]Computing methodologies Neural networks










printfolios=true




: Text-Guided Generation 
 of Functional 3D Environments in Minecraft
    Roberta Raileanu
    
=====================================================================






























§ INTRODUCTION




Procedural Content Generation (PCG) refers to a class of algorithms that can automatically create content such as video game levels <cit.>, 2D or 3D visual assets <cit.>, game rules or mechanics <cit.>, or reinforcement learning (RL) environments <cit.>. PCG allows for compression of information <cit.>, increased replayability via endless variation <cit.>, expression of particular aesthetics <cit.>, and reduction of human labour otherwise required to manually produce artifacts <cit.>. 


These methods are procedural in the sense that they outline sets of procedures or rules for generating artifacts, such as adjacency constraints in Wave Function Collapse, local update rules in cellular automata, or heuristic search in constraint satisfaction.
These procedures often leverage domain-specific knowledge in order to guarantee that generated artifacts are functional; that a game environment does not contain structures that violate physics, or that a player is able to navigate between key points within them.
However, users cannot generally control such methods via free-form language, and control is limited to those metrics explicitly defined by designers.



In contrast, recent generative models have shown impressive abilities in generating diverse images, videos, or 3D scenes from text prompts describing the desired output in natural language <cit.>. These advances allow users to create high-quality content even if they are not domain experts. While these models can produce controllable and open-ended generations, the created content is not guaranteed to be functional. Functionality is particularly important for certain applications such as game design or the creation of RL environments. 

Some recent efforts leverage language models to generate level representations <cit.>, but effectively reduce text-based controls to a series of scalar values.
Other methods train generative text-image models to produce levels that are made of discrete tiles <cit.> or controllable by actions <cit.>, but they do not bring any functional guarantees: houses may spawn in disjointed, and birds may turn into bumblebees after a few frames.


In this work, we pursue a hybrid approach, adapting a generative model to operate on discrete 3D assets and incorporate functional constraints into its loss function.
We propose a new method for generating functional 3D environments from free-form text prompts in the open-world game Minecraft. Our method, , trains a quantized NeRF to produce an environment layout that, when viewed in-game, matches a given text description (see Figures <ref> and <ref> for some examples). 
Experimenting with various quantization schemes, we find that using soft air blocks or annealing them from continuous (soft) to discrete is crucial for learning stability, while using discrete block types leads to the most recognizable structures. We evaluate the fidelity of our approach in matching generated artifacts to descriptions of both generic and domain-specific scenes and objects. We find that  produces in-game artifacts that align with inputs more consistently than a baseline that post-processes the output of an unconstrained NeRF. 

Thanks to its quantized representation of the game world,  can jointly optimize loss terms that enforce local functional constraints on patterns of blocks. We show how this can be instantiated to, for example, generate 3D structures that match a target distribution or obey certain adjacency rules over the block types. 

By inheriting a high degree of expressivity and controllability from the NeRF, while still being able to incorporate functional constraints through domain-specific objectives,  combines the strengths of both PCG and generative AI approaches, representing a first step towards democratizing flexible yet functional content creation. Our method has potential applications in the development of AI assistants for game design, as well as in the production of diverse and controllable environments for training and evaluating RL agents.

To summarize, our paper makes the following contributions:



  * introduces , a new method for training a quantized NeRF to produce 3D structures that match a given textual description using a set of discrete Minecraft blocks,

  * studies different quantization schemes such as whether to use discrete or continuous block densities and types, 

  * shows that the quantized NeRF produces more accurate Minecraft artifacts than an unconstrained NeRF, and

  * demonstrates how to incorporate functional constraints such as obeying certain target block distributions or adjacency rules.







































































§ RELATED WORK


Procedural Content Generation (PCG) is becoming increasingly more popular for training and evaluating robust RL agents that can generalize across a wide range of settings <cit.>. 

Generative models like ours provide a way of biasing the environment generations towards human-relevant ones, thus enabling to search more efficiently through vast environment spaces. Existing works indicate that environment generation can be controlled via computable metrics <cit.>.
Novel environments can also be generated by learning on human datasets <cit.>, sometimes with additional functional constraints or post-processing <cit.>. More recently, <cit.> use large language models to generate Sokoban and Mario 2D levels. However, our work is first to show how multi-modal models can be leveraged to guide the generation of 3D game environments. One of the most popular PCG algorithms is wave function collapse <cit.> which generates structurally consistent content from a single sample, such that its output matches tile-frequency and adjacency constraints. In this paper, we show how such constraints can be used in conjunction with text-guidance to generate environments where both high-level (e.g. via natural language) and low-level (e.g. via block target distributions or adjacency rules) aspects can be controlled.

Text-to-3D Generation Our work builds upon the many recent advances in text-to-3D generation <cit.>. For example, DVGO <cit.> is a supervised NeRF method that, instead of training an multi-layer perceptron (MLP), directly optimizes a voxel grid over the 3D space. 

Similarly, PureCLIPNeRF <cit.> uses a CLIP loss to guide a NeRF using both direct and implicit voxel grids. Our approach, , resembles an implicit voxel grid approach, in that it uses MLPs to parameterize activations over discrete grids. 
But instead of outputting continuous RGB and density values and interpolating between nearest grid vertices (to determine activation at a given point during ray tracing), our approach uses MLPs to produce predictions over block types, and considers only the single nearest grid vertex during ray sampling (to determine within which block a given point resides).


Text-to-Environment Generation
<cit.> train a 
text-conditioned decoder model on a dataset of hand-made levels in a 2D tile domain.
Compared to NeRFs, which train a model to represent a single artifact (here, a level), the “5 dollar" decoder model can represent a distribution of artifacts.
It can also potentially achieve a certain amount of generalization thanks to the pre-trained LLM which is used to encode text prompts.


Minecraft Environment Generation Several prior works have sought to generate Minecraft environments using both supervised and self-supervised methods. <cit.> use a 3D GAN <cit.> architecture to generate arbitrarily sized world snippets from a single example. Meanwhile, <cit.> envision Minecraft as a potential “sketchbook” for designing more photorealistic 3D landscapes, using an unsupervised neural rendering approach to generate the latter from preexisting Minecraft landscapes.

To assist Minecraft players, <cit.> introduce a tool for interactive evolution using both a 3D generative model to generate the structure design and an encoding model for applying Minecraft-specific textures to the structure's voxels. <cit.> have shown that neural cellular automata can be used to grow complex 3D Minecraft artifacts made out of thousands of blocks such as castles, apartment blocks, and trees. Other works have leveraged search-based methods <cit.>, evolutionary algorithms <cit.>, or even reinforcement learning <cit.> approaches to generate Minecraft structures. The game has also been a testbed for PCG algorithms <cit.>, open-endedness <cit.>, artificial life <cit.>, RL agents <cit.>, or foundation models for decision-making <cit.>. However, our work is first to generate functional Minecraft environments directly from text prompts, enhancing the high-level controllability of the creation process. 










































§ : TEXT-GUIDED MINECRAFT ENVIRONMENT GENERATION























 §.§ Quantized NeRFs for Environment Generation


In this section, we introduce , a quantized NeRF which learns to arrange in-game assets during training (see Figure <ref> for an overview of our approach).

Our text-guided NeRF implementation uses score distillation sampling from a pre-trained image generation model to provide a loss function for the optimization of the NeRF.
We use a preliminary version of Emu <cit.>, trained only on Shutterstock data.

Note that our approach is agnostic to the text-to-3D model used, so it can be applied to any other text-to-3D model architecture and thus benefit from future advances in that field. Also note that we do not use any additional or domain-specific data to train our model apart from the block textures in Table <ref>. 



From Continuous Points to Discrete Blocks As in PureCLIPNeRF, we use MLPs to predict continuous activations over a grid by feeding them X, Y, Z coordinates which are encoded using a series of sine waves, as in the original NeRF <cit.>. 
We sample X, Y, Z vertices along a cubic 3D grid of width N, generating predictions over solid block types at each cell in the grid.

We refer to the resulting tensor as the soft solid block grid B_soft∈[0, 1 ]^M × N× N× N, of width N, comprising of M different non-air block types.
We take gumbel softmax <cit.> over these predictions to yield a discrete grid of solid blocks B_hard with values in [0, 1].
We feed the same X, Y, Z coordinates to a separate MLP to generate a soft air block grid A_soft∈[0,1]^N× N× N, and again apply gumbel softmax to generate its discrete counterpart A_hard∈{0, 1}^N× N× N. The distinction between air and solid blocks is analogous to that between the air and albedo MLPs in standard NeRFs.

During training, we may interpolate hard and soft variants to interpolate final air and block grids:

    A= α·A_hard + (1-α)·A_soft      B = β·B_hard + (1 - β)·B_soft,
 where α and β in [0,1] control the hardness of air and solid blocks respectively. In our experiments, we set these to 1 and 0 when air/solids are hard (discrete) and soft (continuous) respectively, and linearly scale between 0 and 1 to “anneal” the blocks from continuous to discrete. We then mask solid block activations with air activations to yield the final block grid C = A ⊙B, with C ∈ [0,1]^M× N× N× N, where ⊙ denotes the element-wise product, broadcasting over the block type dimension in B. Thus, C can be seen as a 3D image with as many channels as there are solid block types. When a cell in C has a value of 1 in a given block channel, and 0s elsewhere, it contains only this block type. When it has values in (0,1), a block is “partially” present. And when a cell in C is all zeros, it represents the presence of an air block. When exporting structures to the game engine or computing loss from functional constraints, we set α=β=1, producing an entirely discrete block grid C_hard.

From 2D Textures to 3D Structures 



The block grid C is combined with statically-generated voxel grids to differentiably generate artifacts that visually resemble in-game structures.
First, we pre-fabricate 16× 16× 16 voxel grids for each block type using in-game textures, applying the game's 16× 16 RGB textures to the appropriate block faces (picking an arbitrary priority order among faces where voxels overlap along the edges of the cube).
These grids are frozen and do not pass gradients during learning.
During the forward pass, we project our M× N × N× N block grid into a 16N× 16N × 16N voxelated block grid, where blocks appear in the same arrangement as in the low-resolution block grid, but in their voxelated form.
We then apply neural ray tracing to this structure to generate 2D images.



When sampling points along a ray during rendering, each point takes the color and density values of the voxel in which it falls, avoiding interpolation between cells to mimic the sharp, pixelated appearance of textures in game. 
To avoid cases where the inside—but not the surface—of a block's voxel grid is sampled during ray tracing, we repeatedly stack face textures within the block, in a pyramidal pattern, to approximate solid objects (Figure <ref>).


Given a vertex x∈ℝ^3 on the solid block grid, we compute a discrete block type, first using an MLP to generate a prediction b_soft∈ℝ^M over M block types, then obtaining a onehot vector b_hard∈{e_1, e_2, …, e_M} using the gumbel softmax function:







    [c]
    b_soft   = MLP(x; θ_B)
        [c]
    b_hard   = gumbel_softmax(b_soft),

where θ_B denotes the parameters of the solid block type MLP. These vectors serve as the elements of B_soft and B_hard respectively.


Again given the block grid vertex x, we compute soft and hard air values, where high values correspond to solid blocks and low values approaching 0 correspond to air.

The soft air grid is computed in the same way as density in previous works, passing the output of an MLP (parameterized by θ_A) through an exponential activation function[During the backward pass, the exponential function is clipped to avoid exploding gradients: σ_soft = exp(clamp(y, 15)).]:

    σ_soft   = exp(MLP(𝐱; θ_A)),

where θ_A denotes the parameters of the air block MLP. To quantize the air grid (effectively computing the presence of air blocks), we first use the soft air values to derive the respective probability of an air/solid block appearing at x:

    [c]
    σ_soft'    = σ_soft - 10
        [c]
    
    p_air   = -σ_soft'
        [c]
    
    p_solid   = σ_soft'.

















We then use the gumbel softmax function to discretize these predictions, obtaining a density value of 0 in case of air, and 1 in case of a solid block:

    σ_hard   = ∑^2gumbel_softmax([p_air, p_solid]).

These values serve as the elements of A_soft and A_hard, respectively.



We use only opaque solid Minecraft blocks (excluding transparent blocks like glass or ice, and porous ones like grass or flowers). 



We use a separate, shallow background MLP, which takes as input a viewing angle, to model color at the end of each ray, allowing the model to learn a low-resolution background texture (effectively projected onto the inside of a sphere). 




 §.§ Functional Constraints
 





Distributional Constraints
The discrete block grid resulting from the quantized NeRF allows us to optimize  to produce a level satisfying a target distribution of block types, producing text-guided objects comprising of particular block mixtures. The user sets a target proportion for each block type, and we apply a loss equal to the difference between this target and the actual proportion of (non-air) grid cells in the NeRF's output (after quantization), which we compute by taking the sum over the relevant channel of the discrete onehot block grid and dividing by the size of the grid. Formally, we define the distributional loss as

    L_D    = ∑_t∈ T|G(t) - P(t)|,

where t is a block type among the set of blocks T, G(t) is the target number of occurrences of this block as specified by the user, and P(t) is the number of actual occurrence in the quantized block grid C_hard.




Adjacency Constraints 
We also introduce a loss term corresponding to a penalty or reward incurred whenever a particular configuration of blocks appear in the generated structure. To this end, we construct a convolutional layer that outputs 1 over any matching patch of blocks, and 0 everywhere else, then sum the result, yielding the number of occurences of the relevant pattern in C_hard. We multiply this sum by a user-specified loss coefficient (negative when the pattern is desired, positive when prohibited). Suppose the user wants to apply a loss/reward of l_p to the pattern of blocks b_0,b_1,…,b_j_p occupying a patch of size K^3 (with the number of blocks of interest j_p ≤ K^3), where each block b_i has relative coordinates x_i,y_i,z_i in the patch. We construct a 3D convolutional weight matrix W_p consisting of the onehot vectors e_i corresponding to each block type, and placing them at position x_i,y_i,z_i in the weight matrix. We apply the resulting convolutional layer, conv_W_p to the quantized block grid, subtract j_p - 1 from the output, and apply a ReLU activation to obtain the binary pattern-activation tensor. Formally,

    L_P    =∑_p∈ Pw_p∑_i^K^3(ReLU(conv_W_p(C_hard) - j_p + 1))_i,

where an inner sum is taken over elements i in the binary activation tensor for pattern p.












§ EXPERIMENTS

Baselines We compare the performance of , our quantized NeRF which learns to arrange in-game assets during training, to , a baseline that maps the continuous outputs to game assets after training.  trains a text-guided NeRF and then maps its output to Minecraft blocks using nearest neighbors. For text-guidance, we use a variant of Emu <cit.> trained only on Shutterstock, a model using text-conditioned latent diffusion to generate images. The nearest RGB value to each Minecraft block is determined by taking the average color over each of the (normally repeated) 16× 16 textures covering each of its 6 faces. We define a width-N grid over the 3D output space of the unconstrained model and query the RGB and density values at the center of each cell in the grid. We calculate the L_2 distance between each centerpoint and average Minecraft block color, mapping each cell to the closest block. We then select a density threshold s=10, and place air blocks wherever σ < s. 


Ablations We study different quantization schemes in order to understand what is the best way to map the continuous outputs of the air and solid block MLPs into discrete grids of Minecraft blocks. The output of either MLP can be passed through the gumbel softmax function to produce a discrete grid of air or solid blocks (see Figure <ref>). If these values are not discretized  α < 1 or β <1, meaning that b_soft or σ_soft are used instead of their “hard” counterparts, then the resulting voxel grid can include solid blocks interpolated with air blocks ( semi-transparent) or with one another ( multi-texture). We also experiment with linearly annealing these values from their soft to hard counterparts over the course of training.
 

Evaluation Datasets We evaluate our method on both generic and domain-specific text prompts, using the COCO <cit.> and Planet Minecraft <cit.> datasets, respectively. For COCO, we use the same 153 prompts as in prior text-to-3D works <cit.>. For Planet Minecraft, we take the names of the top 150 most downloaded assets uploaded by users in 2016 under the “Maps" category. Some examples include “a desperate and lonely wizards tower pmc chunk challenge entry lore”, “mario kartgba bowsers castle 2”, and “icarly set and nickelodeon studio”. A full list of prompts can be found in Section <ref>.


Evaluation Metrics To quantitatively evaluate the performance of our model, we measure its fidelity using R-precision. More specifically, we query other pre-trained joint text-image encoders, namely CLIP ViT-B/16 and CLIP ViT-B/32 <cit.>, and test whether they can recognize the caption responsible for a given NeRF rendering from a set of distractors (other, randomly selected captions from the dataset). For each caption, we repeat the process for 5 different test images and average the result.




























 §.§ Quality of the Generations



In Figures <ref> and <ref>, we can see that, using only Minecraft blocks, our model produces structures that are visually similar to those of the , for both generic and domain-specific text prompts. 

Note that the  model is a strong upper bound because it has a continuous and thus much larger output space (i.e., higher resolution) than our discretized  model.




’s relative performance increases when moving to a set of
domain-relevant prompts.

In Table <ref>, we compare the fidelity of  with that of the  on COCO and Planet Minecraft. Note that here, the generations are evaluated using the renders from the neural ray tracing engine rather than in-game renders. As expected, limiting the NeRF's output to a specific set of discretely assembled blocks drastically reduces its space of generations. This is reflected in 's lower fidelity with respect to the , when generating objects from both the COCO dataset and Planet Minecraft datasets. However, the performance gap between  and the  is reduced when moving from the generic COCO dataset to the domain-specific Planet Minecraft dataset. This suggests that despite its restricted output space,  is particularly effective at generating high-quality structures when the input prompts are relevant to the (discrete)
domain at hand. 









In Table <ref>, we evaluate the R-precision using 2D captures of generated Minecraft block layouts in the game engine itself. Note that here, the generations are evaluated using the in-game renders rather than the neural renders. For game design applications, in-game render fidelity is more relevant than neural render fidelity, so this is ultimately the metric we care about in our study. In this case, the fidelity of the Unconstrained NeRF is lower than that of  for both COCO and Planet Minecraft. This indicates that post-processing the output of a NeRF in order to discretize it using nearest-neighbor leads to worse results than learning to use discrete blocks during the generation process.  

In Figure <ref>, we see that mapping a discrete set of grid vertices to nearest neighbor block types via average color leads to sub-optimal results that are particularly bad at maintaining consistency in terms of texture and color.
This result demonstrates the difficulty of translating unconstrained generations to a constrained repertoire of domain-specific assets.
We conclude that by incorporating these game assets in the learning process, we can generate more faithful in-game structures.



 §.§ Quantization Schemes


In Table <ref>, we compare the effect of applying soft, hard, and annealed quantization schemes to solid and air blocks. Maintaining soft air blocks (continuous-valued block transparency), in combination with hard solid blocks (discrete block types), leads to the highest R-precision at test time.
This suggests that learning the topology (in contrast to the color/texture) of a generated structure is a sensitive process in which relaxing the quantization scheme (and presumably simplifying the loss landscape) is crucial.




Forcing block density to be discrete throughout training leads to poor performance (as can be seen in the first row of Figure <ref>), with the poorest performance coming from models in which both block type and density are fully discrete. 

This may be because of noisier learning dynamics resulting from the quantized output space of the model.  


Conversely, using soft block density can lead to situations in which apparently solid surfaces are emulated by layering a number of semi-transperent blocks. At test time, when rendering the fully discrete block grid, such surfaces can suddenly be culled from the image, as none of the individual blocks of which they consist are enough to result in a “solid” binary output after quantization.

























































§ BLOCK GRID RESOLUTION







We experiment with the resolution of the block grid, learning a N× N× N-block representation of text prompts with N∈{10, 20,…, 100}.

In Table <ref>, we see that increasing block grid resolution leads to an increase in R-precision. We note that the more blocks in the grid, the closer each block comes to being represented by only a single pixel in each 2D render of the 3D block layout. In other words, we can expect the output of these higher-resolution quantized NeRFs to approximate that of their unconstrained counterparts with increasing accuracy. By analogy with visual art, we can say that the model uses blocks in an increasingly pointillistic fashion.




 §.§ Functional Constraints


In Figure <ref>, we illustrate the effect of adding distributional constraints to a prompt asking for “a stylish hat”. We can see that the model produces a similar structure using either entirely gold, redstone, or an even split of both when adding the distributional loss term.

In Figure <ref>, we illustrate the effect of adding an adjacency constraint prohibiting sand from “floating”, i.e. being placed directly above an air tile, and ask for “space needle accurate” (a Planet Minecraft prompt). As the weight of the adjacency loss term is increased, the use of sand blocks becomes restricted to the central “bulb” of the tower, where it is more likely to be supported by a the circular base of dirt blocks. When the adjacency loss is weighted less heavily, sand is often incorporated into the underside of the bulb (where it is unsupported and will fall to the ground in-game).

These experiments demonstrate that functional constraints can be easily integrated with our text-to-3D model to create controllable Minecraft structures that obey both high-level and low-level user specifications and can be rendered in-game. 





§ CONCLUSION


In this work, we develop a new approach for generating functional game environments in Minecraft from free-form text descriptions.  quantizes the output of a text-to-3D NeRF to predict discrete block types which are then mapped to game assets ( voxel-grids corresponding to in-game blocks). This allows the NeRF to use the game assets to represent the content described by a text prompt. We demonstrate that our approach has higher fidelity to the text prompt than a baseline that discretizes the output of an  after learning.  is, to our knowledge, the first generator capable of generating diverse, functional, and controllable 3D game environments directly from free-form text. Since our model can adapt to the unique appearance of user-supplied modular game assets to produce environments with high-level aesthetic properties, it may be particularly useful for game designers working in new domains that don't yet have have large datasets of game layouts. 

One limitation of  is that it takes a few hours to generate a single structure. However, it could benefit from recent and future speed improvements in NeRFs <cit.>. Another promising direction for future work is to model lightning and shadow, in addition to color and density, which could be achieved using an auxiliary loss to model the in-game lightning effects.

While we focus on MineCraft,  could be extended beyond cube-based environments, to any 3D environment involving discrete assets that can be approximated by voxel grids. It could also incorporate more complex functional constraints, assuming these could be implemented to be differentiable. For example, one could compute the path length between key blocks (e.g. a player spawn block or a treasure chest) using convolutions, and use the difference between the target achieved path lengths as part of the loss, similar to the reward used in <cit.>. It may also be beneficial—especially where functioinal constraints cannot be made differentiable—to use RL methods to approximate the gradient from such additional functional scores. Vis-a-vis embodied player agents, environments could be generated to result in certain rewards, dynamics, regret or learnability with respect to these agents.








ACM-Reference-Format






§ MINECRAFT TEXTURES





The set of Minecraft textures used to imitate in-game blocks within the neural rendering engine is displayed in Table <ref>.

































§  GENERATIONS








































































  













































  





§ LIMITATIONS


In some cases, the model uses negative space to represent an object, modulating the background texture to a particular color, then occluding parts of it with foreground blocks/density, to give it an apparent shape. This undesirable swapping of roles between foreground/background models may be more likely to occur in the quantized NeRF: whereas certain colors or textures may be difficult or impossible to replicate using the provided blocks, the background MLP remains unconstrained. To mitigate this, future work could investigate constraining the background MLP to only use 2D projections of “distant” game assets.

Another potential issue is the lack of semantic grounding with respect to block types. For example the model may just as well satisfy the prompt “large medieval ship” by using a combination of dirt and redstone, as with actual wooden logs or planks, so long as these give the appearance of wood. Our preliminary work on functional constraints suggests that this particular problem can be addressed by setting per-block-type targets (e.g. requiring 0% dirt and 50% wood blocks), but a more general approach might lie in “demonstrating” what each block type should be used to represent by adding this information in the prompt.


Whereas traditional NeRFs can model lighting and shadows, this is not the case in , where the color at each point in 3D space is derived directly from a voxel grid corresponding to the in-game appearance of a Minecraft block. When structures are rendered inside the neural engine, they thus appear “flat” in contrast to the kind of shadow and lighting effects that appear in the Minecraft game engine. Ideally, we could train an auxiliary model to mimic the effects of in-game lighting, for example by training it on paired datapoints of 3D block grids, and their appearance in-game at various angles. This learned renderer could replace the differentiable raycasting component of the NeRF pipeline (as in <cit.>), further sparing us from having to re-implement the rendering of irregular game objects such as plants and glass.

 is currently too slow to be feasibly used in an online player-environment generator loop, taking a few hours to generate a single structure. Future versions could benefit from recent and future speed improvements in NeRFs <cit.>. Alternatively, it could be leveraged to generate a training set for a conditional, guidance-free generative models of game worlds.






§ PLANET MINECRAFT DATASET



To test 's ability to generate environments specific to the domain for which it was designed, we source text prompts from Planet Minecraft, a fan-operated site where users can upload and share custom content. We consider a subset of assets uploaded to the “Maps” category in 2016 (the year in which the most such assets were uploaded), and select the top 150 maps of this subset as measued by the number of user downloads. The prompts correspond to the names of these assets. We do not collect the assets themselves or any further data from the site.




The set of prompts scraped from Planet Minecraft is given below:


