{'fraction_non_alphanumeric': 0.03357249553101354, 'fraction_numerical': 0.010896980679285942, 'mean_word_length': 4.5858295718779924, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Start Small: Training Game Level Generators from Nothing by Learning at Multiple Sizes

A procedural level generator is a tool that generates levels from noise. One approach to build generators is using machine learning, but given the training data rarity, multiple methods have been proposed to train generators from nothing. However, level generation tasks tend to have sparse feedback, which is commonly mitigated using game-specific supplemental rewards. This paper proposes a novel approach to train generators from nothing by learning at multiple level sizes starting from a small size up to the desired sizes. This approach employs the observed phenomenon that feedback is denser at smaller sizes to avoid supplemental rewards. It also presents the benefit of training generators to output levels at various sizes. We apply this approach to train controllable generators using generative flow networks. We also modify diversity sampling to be compatible with generative flow networks and to expand the expressive range. The results show that our methods can generate high-quality diverse levels for Sokoban, Zelda and Danger Dave for a variety of sizes, after only 3h 29min up to 6h 11min (depending on the game) of training on a single commodity machine. Also, the results show that our generators can output levels for sizes that were unavailable during training.


Introduction
Procedural Content Generation (PCG) is the process of generating content automatically using algorithms. PCG has been commonly used to generate levels and other assets for games since decades ago. In some games, a larger level size could allow the designer to build a more interesting environment. For example, any 3 × 3 Sokoban [1] level can be solved in 14 steps or less and they are unlikely to pose any challenge to human players. Comparatively, there are many larger Sokoban that demands a higher level of puzzle solving skills and long-term planning. However, training a level generator from nothing tends to be harder as the level size increases. The larger the level, the higher the chance that a bad action is taken during the generation process, rendering the level unplayable. Therefore, if the generator receives no positive feedback except for playable outputs, training becomes challenging due to the sparse feedback.
Some methods such as Procedural Content Generation via Reinforcement Learning (PCGRL) [2], Neural Cellular Automata for Level Generation [3] solve the feedback sparsity issue by reward shaping where game-specific supplemental rewards are added to help the generators learn the functional requirements. However, reward shaping is effort-consuming and requires gamespecific knowledge. We propose a novel approach where the generator starts training on smaller levels, then expands to larger ones as it improves. This approach is based on two assumptions. The first assumption is that it is more likely to generate a desirable level via random exploration if the level size is smaller. The second assumption is that the knowledge gained from learning to generate small levels would be useful to learn generating larger ones. This approach presents multiple benefits: First, it simplifies the process of adapting the generator to new games since it does not require designing supplemental rewards. Second, it produces generators that can output levels for a variety of sizes, including sizes that were unavailable during training (out-of-training sizes). In this paper, we apply this approach to train autoregressive generative flow networks (GFlowNets) [4] from nothing. We also present modifications to diversity sampling [5] to work with GFlowNets and to expand the generators' expressive range. Finally, we present our results for three 2D tile-based games: Sokoban, Zelda and Danger Dave. The results will show that our approach produces high-quality diverse levels at a variety of sizes, while requiring significantly lower training times compared to previous works. At the time of writing, our generators are the only deep generative models to present the ability to generate levels at various sizes for the games we target.
Thus, in this paper, we present the following contributions: 1. Propose a novel approach to train generators from nothing by learning at multiple sizes. 2. Apply our multi-size training approach to train a GFlowNet-based level generator for 2D tile-based games. 3. Modify diversity sampling [5] to be compatible with GFlowNets and to expand the generators' range. 4. Present and discuss the results of the proposed methods on Sokoban, Zelda, and Danger Dave.
This paper is organized as follows: Section 2 briefly mentions the related works, then section 3 presents the proposed methods. In section 4, the experimental setup is detailed, then section 5 presents and discusses the results. Finally, section 6 concludes this paper.

Procedural Level Generation
Many recent works focus on learning level generators from little to no training data. If the dataset is small, different methods to bootstrap a generator using a small dataset were proposed [6,7]. Diversity sampling [5] was proposed to improve the solution diversity of bootstrapped Sokoban level generators. Multiple approaches were proposed to learn a generator from nothing (without any training datasets). Procedural Content Generation via Reinforcement Learning (PCGRL) [2] formulates the level generation problem as a reinforcement learning (RL) task. Generative Playing Networks [8] train the generator from nothing using the feedback from an RL agent, which is trained to solve the generator's output. Adversarial Reinforcement Learning for PCG [9] follows a similar idea except that the generator is also an RL agent. Other approaches include using Quality-Diversity search to learn a diverse set of Neural Cellular Automata [3], and using Neuroevolution to learn iterative level generators [10]. In addition, generators can be trained to imitate other generators such Mutation Models [11] which learn to imitate evolution.
In [10], they showed that their iterative generator can generate levels at out-of-training sizes for the games: Mario and Maze. Our work differs in two ways. First, their network architecture only views a limited window around the target tile, which prevents it from capturing long distance relationships between tiles. This posed no issues for Mario and Maze since the local context is usually enough to pick a good tile. However, for games such as Sokoban, Zelda and Danger Dave, the local context is not enough, and some decisions require knowledge about all the previous decisions. For example, in all these 3 games, there must be only one player tile, so the generator must know if any player has already been added to the level. Hence, we use a recurrent model, inspired by the Long Short-Term Memory used to generate Mario levels in [12], so that it can recall any necessary information about its past decisions. Second, we utilize multi-size level generation to improve the training process even if generating levels at different sizes is not desired.

Generative Flow Networks
Generative flow networks (GFlowNets) [4] learn to build compositional content by applying a sequence of actions. GFlowNets formulates the problem as a directed acyclic flow graph where the generation process starts at an empty object s 0 (the root node) and by applying an action sequence (each denoted by an edge), it reaches a complete object s f (a leaf node). Let z 0 denote the flow entering the graph through the root node, which would branch at nodes and flow though the edges till it pours into the leaves. The goal of GFlowNets is to learn a flow such that each leaf node receives a share equal to its reward R(s f ). Then, GFlowNets can be used as a stochastic policy where the probability of each transition P f (s i+1 |s i ) is proportional to the flow going through its corresponding edge. So, GFlowNets has been used to sample a diverse set of compositional objects such molecular graphs [4], thus they are a good match for procedural level generation.
Out of the loss functions proposed for GFlowNets, we will focus on the trajectory balance loss function [13], which is shown in Eq. (1). It works by matching the forward and backward flow through a given trajectory. The trajectory could be sampled from the current policy or from a dataset. This loss function L requires the estimated source flow z 0 , the reward function R(s f ), the forward policy P f (s i+1 |s i ), and the backward policy P b (s i |s i+1 ). For auto-regressive sequence generation, each state has only one predecessor, so P b (s i |s i+1 ) is always 1.

Multi-Size Generator Training
The multi-size training process trains the generator at multiple sizes in parallel. The sizes should include the desired sizes, a seed size and intermediate sizes to facilitate the generator's transition from the seed size to the desired sizes. The seed size is recommended to be small (close or equal to the smallest possible size). It is also possible to supply multiple seed sizes. Even if some of the supplied seed sizes are too small to contain a playable level, they have no negative effect on the training process, except for a small increase in the training time. Supplying multiple seed sizes should reduce the effort required to find a good seed size.
To generate levels at a variety of sizes, the generator's network must accommodate different level sizes without changing its architecture. So, we formulate the problem as an auto-regressive sequence generation task and use recurrent neural networks (RNN), since they can store and recall important information about all the previous steps. RNNs can generate levels of any size with no architecture change, assuming their memory has enough capacity to store all the relevant information.

Multi-size Generative Flow Networks
A level generator can be trained as a generative flow network without requiring a dataset. We picked a conditional RNN and trained it using the auto-regressive trajectory balance loss function [13]. Therefore, the network needs to learn the source flow z 0 and the forward policy P f . The loss function can be written as shown in Eq. (2).
L is the trajectory balance loss where the trajectory is [s 0 , s 1 , ..., s f ]. u are the control values, w & h are the level's width & height respectively, γ (w,h) are the source flow estimation network's weights for the size (w, h), θ are the forward policy RNN weights and R(s f |u) is the generated level reward given the requested controls. Unlike the forward policy network, the source flow network has different weights for each level size to improve the training stability since the training for each size progresses at a different pace. So, the model cannot estimate the source flow for out-of-training sizes, but it is not an issue since the generation process only needs the forward policy.
Since the probability of generating a level is directly proportional to its reward, it would seem intuitive that an undesirable level's reward R − should be 0. However, the loss function operates in the log space and a zero reward would introduce −∞ to the gradient calculation. So, a non-zero R − value must be picked, while restricting the undesirable levels' coverage of the output distribution to be less than a certain value P . To calculate a bound on R − , we assume that P = 50% (for simplicity, but a smaller value can be picked for a tighter bound) and the level space contains only one desirable level (the worst-case scenario). Since the level space size is |A| wh , where A is the tileset, the remaining |A| wh − 1 levels are undesirable. So, we want to satisfy the bound in Eq. 3 where R + is the desirable level's reward. Assuming R + ≥ 1, a valid value for R − would be |A| −wh . Overall, the reward function in the log-space is shown in Eq. 4 where theû(l) are the actual values of the level properties corresponding to the controls u.
To increase the sample efficiency, an experience replay buffer is populated with any new playable levels generated during training alongside its actual control values (not the controls used to generate it). Populating a replay buffer can be seen as an equivalent to the bootstrapping method proposed in [6]. In addition, the training at non-seed sizes is delayed till the replay buffer is populated with at least one level of the corresponding size, allowing the network to focus on learning at smaller sizes first.
Since the model is conditional, controls must be supplied to the network to roll out new trajectories, but as discussed in [5], the controls have no inherent parameterized distribution from which they can be sampled. In [14], the controllable PCGRL environments sample the controls from a uniform distribution bounded by a user-defined range, so it assumes the user knows the desired bounds beforehand. Also, a tight bound will limit the generator's range, while a large bound may contain many regions of unsatisfiable values, thus decreasing the chances of generating playable levels during training. In [5], it was proposed to learn the distribution of controls from the augmented Experience Replay Buffer

GFlowNet Generator
Level Analyzer output feedback playable levels experience replay sampled conditions Figure 1: The GFlowNet level generator training process. First, the generator is requested to generate output (levels) given a set of conditions sampled from the replay buffer. Then, the analyzer checks the output, send feedback (rewards) to the generator, and add any new playable levels to the replay buffer. The generator is trained using the analyzer's feedback and using levels sampled from the replay buffer (as denoted by the dashed arrow).
dataset. For testing the generators, it was proposed to fit a Gaussian Mixture Model (GMM) on the final augmented dataset. During training, the controls were sampled from a uniform distribution bounded by the minimum and maximum control values in the augmented dataset. It is faster to update compared to GMMs and it will expand as the training proceeds, but such an expansion could lead to sampling more unsatisfiable controls. So, we suggest sampling controls from the replay buffer, then adding some noise, to explore the regions surrounding the current generator range. To avoid bias towards the replay buffer modes, we suggest sampling conditions via diversity sampling. If the replay buffer for a certain level size is empty, the controls are sampled from the closest populated buffer based on level size (argmin i |wi − w| + |hi − h|). The same criterion is applied to a pick a GMM for generating levels with out-of-training sizes. If the replay buffer is empty for all the sizes, we assign any random numbers to the conditions. To sum up the interactions between the different components, Figure 1 shows a block diagram of our GFlowNet level generator training process. Since random noise is added to the sampled controls, the values could end up invalid (e.g., 2.5 crates). Even if the network may have no problem generating levels from invalid control values (usually true if they are close to the generator's range), they will cause an issue while deciding the level reward log(R(l|u)). A solution is to snap the sampled controls to the nearest valid values. For example, a sampled crate count for Sokoban could be rounded to the nearest integer then clamped to the range [1, wh − 2]. Clamping is optional, especially when the bounds are unknown. Having a few unsatisfiable controls during training did not cause problems in our experiments.
In [5], the controls were level properties divided by functions of the level size as a replacement for input normalization. The same idea is applied here to improve the usability of condition models across sizes. For example, generating a level with a 250-step solution is unlikely if the level is 5 × 5, but more likely if it becomes 7×7. So, the GMM should not learn that a 250-step solution is unlikely. If the solution length is divided by the level area, the GMM would learn that generating a level, whose solution length is 10× the level area, is unlikely regardless of the size. This presents no generalization guarantees, especially since the denominators were picked by intuition in our experiments. While trying to tune the denominators, no notable performance differences were observed when generating at in-training sizes, but they had some effect for out-of-training sizes. Still, creating a tailored GMM for the targeted size could improve the results. In that case, a two-step process can be followed: generate a level sample at the targeted size using controls sampled from the GMM for the closest in-training size, then fit a GMM using the playable portions of the generated sample. In our experiments, using the tailored GMM has always increased the probability of generating a playable level, and for some games, it also improved the controllability. But for Sokoban, the diversity dropped, since the tailored GMM was tighter than the GMM for the closest in-training size. So, we use tailored GMMs for out-of-training sizes with some games only, as will be stated in section 4.2.

Diversity Sampling and Reward
Diversity sampling was proposed [5] to increase the solution diversity of Sokoban generators trained with bootstrapping [6]. Since our generator is trained from nothing, the initial replay buffer population will likely have a limited diversity, so diversity sampling is crucial for our method. When diversity sampling is applied, the levels are clustered based on their properties and training batches are collected by sampling each level uniformly from a uniformly sampled cluster. So, the probability P div of sampling a level l is as shown in Eq. (5) where C key(l) is the cluster containing l and N is the number of clusters. When auto-regressive models are trained using the crossentropy loss, the output distribution follows the training data distribution. In GFlowNets, however, the output distribution follows the reward distribution. So, we add a diversity reward R d iv ∝ P div . Since the reward function in Eq.
(4) requires that R + ≥ 1, the diversity reward is picked to be as shown in Eq. (6) where {C1, ..., Cn} are the clusters. Despite the diversity reward, diversity sampling is still needed to ensure that rare levels appear frequently during training.
In [5], the levels were clustered based on a distilled form of the solution, called the solution signature. However, the solution signature proposed in [5] is only applicable to Sokoban. And, as the results will show, the levels could have a large variety of signatures while only covering a small range over the pushed crate count and the solution length. So, we changed the clustering key to be a tuple containing a set of properties. For example, we picked a tuple of the pushed crate count and the solution length as the cluster key for Sokoban. Additionally, a granularity can be picked to group together close property values. For example, we divided the solution length by (w + h) then floored it before adding it to the cluster key, so that minor solution length differences are ignored. Thus, a level with a unique solution length would not be considered unique, if it is in the vicinity of levels with the same pushed crate count and close solution lengths. Deciding the cluster key properties and granularities is left for the users to pick based on their preferences.
An optional addition is the property reward R prop where the generator is rewarded based on the generated level's properties. For example, if difficult levels are more desirable, a reward proportional to the level difficulty could be added. Overall, the reward function log(R total (l|u)) used in our experiments is as shown in Eq. (7). It is noteworthy that the diversity and the property rewards are added even when the generated level does not satisfy the controls. So, a rare and/or a generally desirable level is still less preferred to a level that satisfies the controls, but more preferred to other more common levels.
GRU Cell input A remaining issue is that obvious tile patterns may appear in a significant portion of the generated levels, despite containing a diverse variety of solutions and properties. This issue can be solved using data augmentation where the levels are randomly flipped (vertically and/or horizontally depending on the game rules) during training.

Model Architecture and Hyperparameters
All the experiments use the model architecture shown in Figure 2. The hidden size of each GRU cell is 128. The conditional embedding module consists of 2 feed-forward layers: 16 neurons (Leaky ReLU) and 32 neurons (No activation). The action module also consists of 2 feed-forward layers: 32 neurons (Leaky ReLU) and |A| neurons (Softmax) where A is the tileset. The last layer's weights and biases are initialized to zero. A source flow estimator is instantiated for every level size, each consisting of 2 feed-forward layers: 32 neurons (Leaky ReLU) and 1 neuron (No activation). The last layer's weights and bias are initialized to 0 and the output is treated as log(z 0 ). The model traverses the level in a row-wise snake-like pattern (each row is traversed in the reverse order of the previous row).
The models are trained using RMSProp [15] with a learning rate of 0.001 except for the source flow estimators whose learning rate is 0.01. The batch size is 32. All the experiments are run for 10, 000 iterations.
After training, the final experience replay buffer is used to fit a GMM from which the controls are sampled during testing. We use a Bayesian Gaussian Mixture Model [16] with 16 components. The weight distribution prior is a Dirichlet process with α = 1/16, the mean prior is N (E(X), I) where X is the training data and the covariance prior is a Wishart distribution. The GMM is fit for 100 iterations starting from 8 different initializations and the model with the highest lower bound value on the likelihood is picked.
To test a generator, multiple output samples are generated. To examine the output quality and diversity, 10, 000 levels are generated where the controls are sampled unconditionally from the GMM. To examine the controllability, a level sample is generated for each control where the tested control is fixed, and the others are sampled conditionally from the GMM.
The sample size and the tested control values will be stated as we define each control in section 4.2. The test setup for Sokoban matches the setup in [5] to facilitate comparing our method with the methods included in their study. To test the training stability, we run each experiment 3 times (following recent works [2,7]), compute the results for each run separately, then report the mean and the standard deviation. All the experiments are run on the same machine which contains an 8-core 3.7 GHz CPU (16-threads) and an Nvidia RTX 3070 Laptop GPU.

Games
The methods are tested on Sokoban, Zelda and Danger Dave. In this subsection, we briefly introduce the three games, their functional requirements, the in-training & out-of-training level sizes, the controls, the cluster key and the property reward. For each control, we explain it, state its denominator den (used to divide the property when supplied to the network and the GMM fitter), the noise z s added after sampling during training, and the test sample defined by the sampled level count n test for each control value in the

Sokoban
Sokoban [1] is a top-down puzzle game where the player pushes crates around the level until each crate is located on a goal tile. The player can only push one crate at a time and cannot push a crate into a wall or another crate. The functional requirements are: there is only one player, the crate and goal counts are equal, at least one crate is not on a goal, and the solver can win the level. Figure 3 shows a Sokoban level example and a possible solution for it. The generator is trained on the sizes 3 × 3, 4 × 4, 5 × 5, 6 × 6 and 7 × 7. During testing, the sizes 8 × 8 and 9 × 9 are added. The solver uses Breadth-First Search with an iteration limit of 5 × 10 2 for 3 × 3 levels, 5 × 10 4 for 4 × 4, 5 × 10 5 for 5 × 5 and 10 6 for the larger sizes. For data augmentation, the levels can be flipped vertically, horizontally or both. The controls are as follows: 1. Pushed Crates: the number of crates that the solver pushed to solve the level. Since the generator can add a crate and a goal on the same tile, some crates may not need to be moved solve the level. These crates are not counted since they are easy to add and remove without affecting the level solution. den = (w + h)/2, z s ∼ U [−1, 1] crates, n test = 1, 000 and C test = [1,10]. The cluster key consists of the pushed crate count and the solution length with a granularity of 1 and (w + h) respectively. The property reward is the logarithm of the solution length. Controls for out-of-training sizes will be sampled from the GMM of the closest in-training size, since using tailored GMMs significantly decreased the diversity.

Zelda
The GVGAI [17] version of Zelda [18] is a top-down turn-based adventure game where the player has to fetch a key, then exit via the locked door. The level contains enemies that kill the player if they enter the player's tile. The player can attack an adjacent tile to kill the enemy in it. The functional requirements are: there is only one player, key & door, the enemy count is in [1, max(w, h)], there is a path from the player to the key then to the door, and every enemy in the level can reach the player. Figure 4 shows a Zelda level example and a possible solution for it. The generator is trained on the sizes 3 × 4, 3 × 6, 5 × 4, 5 × 6, 7 × 6, 5 × 11 and 7 × 11. During testing, the sizes 6 × 10, 10 × 6, 8 × 12 and 9 × 13 are added. For data augmentation, the levels can be flipped vertically, horizontally or both. The controls are as follows: 1. Nearest Enemy Distance: the shortest path length to the nearest enemy. den = wh, z s ∼ U [−2, 5] steps, n test = 100 and C test = [1, wh/2 ].
(a) t = 0 The cluster key consists of the nearest enemy distance and the path length with a granularity of (w + h)/2 for both. The property reward is the logarithm of the path length. Tailored GMMs will be created for out-oftraining sizes.

Danger Dave
Danger Dave [19] is a platformer game where the player has to fetch a key, then exit through the locked door. Unlike Zelda, this game has a side view, so the player is pulled down by gravity. The level has spikes which kills the player upon touch. The level also contains diamonds which can be optionally collected. The player can move sideways and jump. The functional requirements are: there is only one player, key and door, the player is initially on a ground, the spike count is less than (w − 1) h/2 , the diamond count is in [1, max(w, h)], all the diamonds are reachable (but similar to the original game, some diamonds may be located at positions where the player would die or get stuck after reaching them), and the level can be solved. Figure 5 shows a Danger Dave level example and a possible solution for it. We use the same sizes as Zelda. For data augmentation, the levels can only be flipped horizontally. The controls are as follows:  The cluster key consists of the jump count and the solution length with a granularity of 1 and (w + h)/2 respectively. No property reward was used for this game's experiments. Tailored GMMs will be created for out-oftraining sizes.

Results and Discussion
This section presents and discusses the experimental results. The results explore the method performance regarding the output quality, diversity and controllability. Finally, the training and inference times are reported. Table 2, Table 3 and Table 4 present the results of the Sokoban, Zelda and Danger Dave generators, respectively, trained with a variety of configurations. The tables contain the following metrics:

Quality and Diversity
• 'Playable%' presents the playability which is the percentage of generated levels that satisfy the functional requirements. • 'Diversity' is the tile-diversity which is the average tile-wise hamming distance between all pairs of playable levels divided by the level area.
• 'Unique Sig.' is 100% × the unique signatures count / the playable levels count. This is available for Sokoban only.
• 'Solution Length' presents the mean and standard deviation of the solution lengths of all the unique levels extracted from the 3 runs. To save space, this is reported for Sokoban only.

DS, PR & AUG stands for Diversity Sampling, Property Reward and Data
Augmentation respectively. The Sokoban experimental results include two variation of diversity sampling: one using the solution signature as the key (Signature Key for short) which is marked by 'Sig.' in the table, and one using a tuple (as defined in section 4.2) as the key (Tuple Key for short). Sizes 3 × 3 and 4 × 4 were omitted from the results to save space. As shown in Table 2, the playability statistics are mostly > 50% for the in-training sizes 5 × 5, 6 × 6 & 7 × 7. Training without diversity sampling yields the highest playability and high tile-diversity too. However, the solu-  tion diversity is low as shown by the unique signature and the solution length statistics. Diversity sampling significantly improves the solution diversity at the cost of a significant playability decrease. Although the signature key yields high unique signatures, the solution lengths only cover a small range. Using the tuple key significantly increases the solution length range despite decreasing the unique signatures. Adding a property reward further increases the solution length range. Data augmentation increases the tile diversity and unique signatures but decreases the playability and the average solution length. The duplicate percentage was omitted since it is always 0%, except when using the tuple key without data augmentation, where the duplicates were nearly 1.4%. So, data augmentation also decreases the duplicates. Figure 6 shows the expressive ranges of the Sokoban generators where the x-axis is the solution length, and the y-axis is the number of pushed crates. The results show that the tuple key significantly expands the range and adding the property reward further expands the range along the solution length axis. In Table 3 and Table 4, sizes smaller than 7 × 11 were omitted to leave space for the out-of-training sizes. The results support the claim that data augmentation increases the tile diversity. It also shows that data augmentation does not necessarily decrease the playability as observed in the results for Zelda. Figure 7 shows generated level samples where the levels for each game are sampled from the same model. To show the extent of the generator's range, the sample contains the minimum, median and maximum along the controlled properties. The figure contains an out-of-training size for each game to demonstrate the generators' diversity for out-of-training sizes. Table 2, Table 3, Table 4 and Figure 7 show that our method can generate levels for out-of-training sizes, however the performance can decline, as observed in the results for Sokoban and Danger Dave. In Sokoban, the decline is more prominent when the tuple key is used, which could suggest that the generator was unable to generalize the knowledge it gained for generating levels with more crates and longer solutions to out-of-training sizes.   Table 5, Table 6 and Table 7 present the control statistics of the Sokoban, Zelda and Danger Dave generators, respectively. The tables contain the following metrics:

Controllability
• 'Playable%' presents the playability which is the percentage of generated levels that satisfy the functional requirements.
• 'Avg. Error' is the mean absolute error between the requested and the generated levels' properties which is computed for the playable levels only.
• 'R 2 ' is the coefficient of determination where the request controls are treated as the true values and the generated level properties are treated as the predicted values. It was added because the average error is hard to compare without considering the variance in the control values.
• 'Score' is the control score [5] where the tolerance is 2 for the pushed crate count and 10 for the solution length. This is reported for Sokoban to facilitate the comparison of our results with the ones in [5]. We did not report the control score for the other games to save space.
In Table 5, for the in-training size 7 × 7, data augmentation decreases the playability and increase the average error, thus decreasing the score. The  effect is different for the out-of-training sizes, where data augmentation improves the results in most cases. Regardless, the performance still declines when the requested size is out-of-training. The decline can cause the R 2 values to be close to (or less than) zero. In addition, the standard deviations of all the metrics are usually higher for out-of-training sizes, which means that the results tends to highly vary across runs. In Table 6 and Table 7, only the results with data augmentation were reported to save space, since they were better in most cases especially at out-of-training sizes. The results show some performance decline at out-oftraining sizes, but it is not as significant as seen in the results for Sokoban, which could be attributed to the use of tailored GMMs. Some out-of-training sizes have better results compared to the reported in-training size. In addition, the average R 2 values are rarely near or below zero. An interesting observation is that the best control results are for tile-count oriented controls (Pushed crates in Sokoban, Enemies in Zelda and Spikes in Danger Dave). This can be attributed to the recurrent models' ability to keep count of items in its memory.

Training and Generation Time
All the experiments use the same model architecture and run for the same number of iterations, but the training time differs due to other factors: the number & the area of the training sizes, and the time cost to analyze the generated levels. For Sokoban, training with diversity sampling using the signature key requires the highest training times (7h 36min on average) since most of the generated levels pass the basic checks (e.g., having only one player) and require invoking the solver. On the other hand, training without diversity sampling took 3h 8min on average, since most of the generated levels were trivial to solve. In-between is training with the tuple key, which took 6h 11min on average. For Zelda and Danger Dave, training took on average 3h 35min and 3h 29min respectively, which is shorter than training for Sokoban, because their functional requirements are faster to check.
During generation, the model call is divided as follows: 1 conditionalembedding module call, followed by wh calls to the recurrent cells and the action module. So, the generation time should be directly proportional to the level's area. By timing 1000 generation requests (batch size = 1) at different sizes and applying least squares linear regression, the generation time turned out to be 0.79wh + 0.39 ms on average (r = 0.99999).

Discussion
In general, the results show that the multi-size GFlowNet level generator learns to generate diverse levels for multiple sizes, even for out-of-training sizes. The results also show that the user can control the generator's output.
The closest recent work to our method is Controllable PCGRL (C-PCGRL) [14] since it is controllable and learns from nothing. C-PCGRL's results for 7 × 7 Sokoban level generation are reported in [5], so we will compare it with our results. Regarding the playability, C-PCGRL achieves a higher percentage (70.1%) compared to our method. However, we argue that it is better to compare generators with different generation times using the time needed to generate a playable level. To generate a playable level, C-PCGRL requires 1.43 trials × 617.6 ms/trial = 883.17 ms on average, while our method requires 1.85 trials × 39.1 ms/trial = 72.34 ms only on average. This computation ignores the time to verify a level which differs across games and depends on the level difficulty. Unlike our method, C-PCGRL requires the solver's feedback during generation, and we found that it invokes the solver 14.9 times per episode on average. Thus, our method would still require less time since it only needs 1 invocation/trial. Regarding diversity, both methods are on-par in many aspects. Our method's tile diversity results are close to C-PCGRL's tile diversity (0.52). Similarly, our unique signature percentage is close to C-PCGRL (45.3%). However, our method exhibits a broader expressive range. While C-PCGRL expands to 7 pushed crates and 164-step solutions, our method consistently has a range that expands to 14 pushed crates and 280-step solutions when diversity sampling, property reward and data augmentation are utilized.
Regarding controls, our method has lower mean absolute errors on every control compared to C-PCGRL (2.46 for the crate control and 22.40 for the solution length control). Our method also exhibits higher control score for the crate control (where C-PCGRL achieves 19.3%) and can achieve higher scores for the solution control (where C-PCGRL achieves 17.0%) if data augmentation is disabled. The control score can be improved by doing multiple trials to increase the chance of generating a playable level.
Regarding the training time, C-PCGRL required 66 hours [5], while our method only requires 6h 11min on average (< 10% of C-PCGRL's training time). It should be noted that the machine used in [5] is faster than our machine (based on our replication of their LSTM experiments, our machine requires 125% of their reported training times). There are multiple reasons that our generator is faster to train. First, the experience replay buffer accelerates our training process by increasing the sample efficiency. Second, a trajectory in our method is always wh steps, while it could reach (wh) 2 steps in C-PCGRL. To check if the long training time of C-PCGRL is necessary, we ran a C-PCGRL experiment on our machine where training was stopped at the first checkpoint after 10% of the total time (stopped at 6h 37min). Given only such a short training time, the playability of C-PCGRL had degraded from 70.1% to just 17.2%, and the expressive range shrinks to be up to 80-step solutions only.
Other than the results, C-PCGRL and our method differ in some other aspects. C-PCGRL solves the feedback sparsity problem by reward shaping while our method relies on multi-size training, since the feedback is denser at smaller sizes. Our approach shifts the effort from reward shaping to size picking, which should require less effort. C-PCGRL motivates diversity by tasking the agent to fix a randomly generated level and satisfy the controls within a limited number of changes. Our method motivates diversity by adding noise to the sampled training controls, and by using diversity sampling. Unlike our method, C-PCGRL is compatible with mixed initiative co-creation, since it is an iterative generator [20].

Conclusion
This paper presents a novel approach to train level generators from nothing by learning at multiple sizes. We use an auto-regressive RNN to generate levels are different sizes without changing the network architecture, and train it as a GFlowNet. The motivation is to mitigate the feedback sparsity by training the generator at smaller sizes, where the feedback is denser, in addition to the desired sizes. Additionally, the generator learns to generate levels at various sizes. The results show that our method can generate diverse levels based on user-supplied controls for a variety of sizes (including out-oftraining sizes) and a variety of games (Sokoban, Zelda and Danger Dave). Compared to previous works, our method presents an advantage by having shorter training and generation times, and exhibiting low control error. And, by using diversity sampling with tuples of level properties as cluster keys, the expressive range significantly expands.
Our method has some aspects that would benefit from further research. It would be beneficial to find methods that would improve the model's generalization to out-of-training sizes. Also, finding better methods to sample conditions for out-of-training sizes is left for future work. Our method still requires the user to pick some game-specific training configurations such the cluster key and the seed size, so finding a compatible game-agnostic diversity scheme would be beneficial. Finally, looking for techniques that could apply GFlowNets in a mixed-initiative level design setting is left for future work.