One Noise to Rule Them All: Learning a Unified Model of Spatially-Varying Noise Patterns
Arman Maesumi Dylan Hu Krishi Saripalli Vladimir G. Kim Matthew Fisher Sören Pirk Daniel Ritchie



]:

Learning a Unified Model of Spatially-Varying Noise Patterns


arman.maesumi@gmail.com
0000-0001-7898-8061

  Brown University
  Providence
  RI
  USA



Both authors contributed equally to this research.
dh@brown.edu
0009-0001-9456-9470

[1]
krishi_saripalli@brown.edu
0009-0000-5224-2632

  Brown University
  Providence
  RI
  USA



vova.g.kim@gmail.com
0000-0002-3996-6588

  Adobe Research
  USA



techmatt@gmail.com
0000-0002-8908-3417

  Adobe Research
  USA



soeren.pirk@gmail.com
0000-0003-1937-9797

  CAU
  Kiel
  Germany



daniel_ritchie@brown.edu
0000-0002-8253-0069

  Brown University
  Providence
  RI
  USA





Procedural noise is a fundamental component of computer graphics pipelines, offering a flexible way to generate textures  that exhibit “natural” random variation. 
Many different types of noise exist, each produced by a separate algorithm.
In this paper, we present a single generative model which can learn to generate multiple types of noise as well as blend between them.
In addition, it is capable of producing spatially-varying noise blends despite not having access to such data for training.
These features are enabled by training a denoising diffusion model using a novel combination of data augmentation and network conditioning techniques.
Like procedural noise generators, the model's behavior is controllable via interpretable parameters plus a source of randomness.
We use our model to produce a variety of visually compelling noise textures.
We also present an application of our model to improving inverse procedural material design; using our model in place of fixed-type noise nodes in a procedural material graph results in higher-fidelity material reconstructions without needing to know the type of noise in advance. Open-sourced materials can be found at https://armanmaesumi.github.io/onenoise/https://armanmaesumi.github.io/onenoise/




<ccs2012>
<concept>
<concept_id>10010147.10010371.10010382.10010384</concept_id>
<concept_desc>Computing methodologies Texturing</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257.10010293.10010294</concept_id>
<concept_desc>Computing methodologies Neural networks</concept_desc>
<concept_significance>300</concept_significance>
</concept>
</ccs2012>


[500]Computing methodologies Texturing
[300]Computing methodologies Neural networks




  
    < g r a p h i c s >

  Our method enables the synthesis of a wide range of noise patterns with spatially-varying characteristics. Here we show the flexibility of our unified noise model, allowing one to art-direct their noise in a granular fashion. Our model creates semantically meaningful interpolations between noise configurations; above we see the Siggraph logo written with  that are nested inside of  striations – the scale and distortion of the steel pattern naturally interpolates into a denser pattern before transitioning into fibers. We also show renderings of a clay shader that incorporates our spatially-varying noise patterns. The first three images make use of class-interpolated noise, the final image uses parameter-interpolated noise. Please zoom into the figures for full visual detail.
  


20 February 2007
[revised]12 March 2009
[accepted]5 June 2009

[
    Daniel Ritchie
    Received March XX, 2024; accepted XX XX, 2024
=================================================





§ INTRODUCTION

Procedural noise has long been a fundamental building block in computer graphics, serving as a versatile tool for modeling fine, naturalistic details in a range of applications. It finds widespread use in creating albedo textures, bump and normal maps, terrain height fields, and density fields for volumetric phenomena such as clouds and smoke. There exists a zoo of algorithms for generating such noise patterns, for instance Perlin, Worley, and Gabor noise <cit.>, along with systems for composing and transforming these noise patterns to synthesis more complex shaders (e.g. Blender, Adobe Substance 3D Designer, and other material editors <cit.>).

Despite the advancement of these tools, a fundamental limitation remains in the design process: the necessity to make discrete choices regarding the types of noise to employ. This constraint poses challenges in creative design, where the ideal noise type may not be immediately present among the provided “zoo”, or where desired visual characteristics fall between the behaviors of existing noise types. Designers might envision patterns exhibiting different noise characteristics in various regions, transitioning smoothly without abrupt or artificial blending. The standard technique for blending between noise types—alpha blending—often yields unsatisfactory results, where features do not naturally interpolate in the image (see Figure <ref>).

This limitation also complicates inverse design tasks, such as recovering a procedural representation (i.e. a material node graph) of a texture from observed data. 
Such graphs often use multiple noise generator nodes; if the types of these nodes aren't known in advance, the search space for the inverse design problem grows combinatorially, and the need to search over discrete possible noise types limits the applicability of gradient-based optimization.

In this paper, we address these challenges by introducing a method for learning a continuous space of spatially-varying noise patterns from data that lacks any spatially-varying observations. We leverage a denoising diffusion probabilistic model (DDPM) <cit.> equipped with a spatially-varying conditioning module, which is trained via a novel application of CutMix data augmentation <cit.>. Our method enables the generation of noise that can vary smoothly across the canvas, offering a greater level of control over the resulting noise pattern.
Just as with traditional procedural noise generators, the model can be controlled by changing a set of interpretable parameters or by changing the input source of randomness (i.e. random seed).
Importantly, our method can generate noise images at any requested image size (beyond the size of the training data) and can also produce seamlessly tileable images.

We evaluate our model by using it to generate a variety of  noise blends and spatially-varying noise patterns (including spatial variation driven by image-based masks).
We also present a user interface for painting spatially-varying noise patterns, videos of this interface are included in the supplemental material.
Finally, we show an application of our method to inverse procedural material design, demonstrating that using our model as a noise generator node in a material graph to be optimized can result in higher-fidelity material reconstructions without needing to know the particular noise type for that node in advance.

In summary, our contributions are:

    
  * A generative model which can learn to generate spatially-varying noise patterns.
    
  * A training scheme for the model using a novel application of CutMix augmentation, allowing the model to learn to generate spatially-varying noise patterns without having any spatially-varying training data.
    
  * An application of our model to inverse procedural material design, showing improved material reconstructions without pre-specified discrete noise types.






§ RELATED WORK



  
Procedural noise
“Noise is the random number generator of computer graphics” <cit.>. Procedural noise functions collectively form a body of techniques for algorithmically synthesizing patterns that mimic the randomness and irregularities in natural phenomena. These noise functions have become foundational to many applications within computer graphics, facilitating the synthesis of natural materials <cit.>, terrain <cit.>, and motion <cit.>.

There are a wide variety of noise functions that synthesize distinct patterns, for example, Perlin <cit.>, Wavelet <cit.>, and Gabor noise <cit.>. Such noises are often composed and transformed using various primitive operations (e.g. scalar functions and domain warping), forming patterns that are colloquially referred to as noise in computer graphics — while these resulting patterns may not strictly fall under the formal definition of noise <cit.>, we continue to use the term in this established, albeit technically imprecise sense, throughout the paper. A selection of various noise patterns is shown in Figure <ref>. For further background, we refer to the survey by Lagae et al. lagae2010noiseSurvey.



  
Neural parametric texture synthesis
As noise is a central building block for textures, our work is related to methods for texture synthesis.
Texture synthesis has a deep, decades-long history, a full discussion of which is outside the scope of this paper.
The most relevant methods to our work are parametric texture synthesis methods which use neural networks as their parametric model.
Most of this prior work focuses on example-based texture synthesis, i.e. generating textures which are similar in some statistical sense to an input exemplar.
Some methods do this by optimizing an image such that the features produced by passing it through a certain pre-trained neural network match those produced by the exemplar image <cit.>.
Others train feedforward neural networks to directly generate texture similar to an input exemplar <cit.>.
The closest work to ours is a generative adversarial network (GAN) that does not perform example-based synthesis, but rather learns a continuous latent space of textures <cit.>.
This model supports interpolation between textures, but it is not controllable by a combination of interpretable noise parameters and a source of randomness (its latent space entangles texture “style” with spatial randomness).
We initially experimented with a GAN-based architecture and found that it struggled to capture a data distribution with the many different modes represented by a set of highly distinct noise types.
Prior work suggests that diffusion models such as ours fare better at modeling such distributions <cit.>. 

We also acknowledge the role of non-parametric methods in this field. These approaches synthesize textures by using the structures present within an image, without relying on a learned parametric model. For instance, Matusik et al. simplicialComplex frame the texture interpolation problem through morphable models <cit.>, and define a space of interpolatable textures via a simplicial complex induced by a texture similarity metric. ImageMelding <cit.>, on the other hand, proposes a patch-based method that smoothly “in-fills” regions of a texture via a regularized screened Poisson equation. For a more comprehensive review of this body of work, we refer to the survey by Raad et al. raad2018textureSurvey.



  
Inverse procedural material design
Recovering procedural representations of materials from observations (i.e. photographs) has become an increasingly promising area of research. Material graphs—the procedural representation of choice—feature nodes that generate, transform, and filter various signals that ultimately become spatially-varying material maps (SVBRDFs). Authoring and editing these material graphs is not only time consuming, but also requires mastery of complex software <cit.>. Inverse procedural material design aims to alleviate this difficulty by recovering a material graph from a given image. Recent works tackle this problem in various ways, either by 1) directly predicting the parameters of a given material graph <cit.>, 2) optimizing the continuous parameters of the graph via gradient-based optimization <cit.>, or 3) by synthesizing a material graph from scratch <cit.>.

Most relevant to our work are learned differentiable proxies, which serve as continuous relaxations for gradient-based optimization strategies to the inverse design problem. Hu et al. hu2022diffProxy train several GAN-based generative models to act as differentiable proxies for pattern generators (i.e. a brick tiling generator node). However, these proxies are deterministic, meaning they cannot model noise functions, and critically, each pattern is represented by a separate generative model. Our unified noise DDPM is able to simultaneously capture many non-deterministic noise functions in the same continuous space, acting as a relaxation over an entire space of unique noise functions.







§ METHOD

Given a collection of procedural noise textures sampled from various parametric noise generating functions, we seek to learn a conditional generative model that captures the noise textures given their accompanying parameter configurations, enabling the synthesis of a wide range of noises from a single universal function. Our conditional generative model will be formulated in a way that facilitates synthesizing spatially-varying noise textures (i.e. spatial blends between the categorical noise type as well as noise parameters), even though the training collection does not contain such samples. That is, we assume our given collection of noise samples only exhibits globally uniform semantic properties, as shown in Figure <ref>.

We make use of a denoising diffusion probabilistic model (DDPM) that incorporates a spatially-varying conditioning mechanism. The model is trained using a data augmentation scheme that regulates the behavior of this conditioning module, ensuring that it behaves appropriately when given granular conditioning signals. In the following sections, we first detail the spatially-varying conditioning module, followed by the aforementioned data augmentation strategy. We defer details pertaining to our DDPM training to Section <ref>.



 §.§ Spatially-varying conditioning

At training time, our conditioning signal is given by two vectors, f_c, a class embedding that encodes the categorical label of a particular noise type, and f_p, a list of parameters that determine semantic properties of the generated noise image (i.e. , , etc.). Providing these parameters as explicit input to the model allows interpretable user control; it also disentangles control over the “style” of the generated noise (via these parameters) from the “seed” (i.e. stochastic component) of the noise functions via the initial Gaussian noise provided to the DDPM.

To facilitate synthesis of noise maps with spatially-varying properties, we employ spatially-adaptive denormalization (SPADE) <cit.>. We first map the class and parameter vectors into a shared feature space with a small MLP, call the resulting feature vector f. The feature vector is tiled into a spatial grid, F, of shape |f| × H × W, where H, W are the height and width respectively. This feature grid is used as input to the SPADE module. Following Dhariwal et al. beatGANs, we modulate the intermediate feature maps of our network's Group Normalization layers, making the SPADE conditioning function


S(h, F) = γ(F) ⊙GroupNorm(h) + β(F) 

GroupNorm_g(h) = h_g - μ_g/√(σ_g^2 + ϵ)


where γ,β are convolutional layers that transform the feature grid into element-wise scales and shifts respectively, and h is the incoming activations of the previous layer. The scales and shifts act on the output of a Group Normalization <cit.> block, which is computed using the mean, μ, and standard deviation, σ, of each group of channels g.

Our training data only contains noise textures that exhibit spatially uniform properties, meaning that all feature entries in the grid F are identical at training time. However, at inference time we may query the network with an artificially constructed feature grid – for instance, we can spatially blend between multiple feature vectors to produce a conditioning signal that smoothly interpolates between noise types and noise parameters, as shown in Figure <ref>.


  §.§.§ Spherical class embeddings [The following subsection introduces an enhancement to our original method, identified post-publication, which improves results in many cases.]

As previously mentioned, the network is conditioned on a class embedding, f_c, that is learned for each noise type during training. We find that regularizing this embedding space leads to substantial improvements when interpolating between classes. In particular, we penalize the deviation of the norms of these embeddings from a target norm. For embedding vectors of dimension d that are initialized from f_c∼𝒩^d(0,I_d), we define the target norm, T, as the expected squared norm of a d-dimensional Gaussian vector according to the identity,

    T_d^n 𝔼_ f_c∼𝒩^d(0,I_d)[||f_c||_2^n] = 2^n/2 Γ((d+n)/2)/Γ(d/2),

where Γ denotes Euler's gamma function. During training we then penalize the embeddings according to 

    ℒ_reg = 1/|C|∑_c∈ C(f_c_2^2 - T_d^2)^2,

this imposes a spherical structure on the embeddings, which means that interpolating between them can be done via spherical linear interpolation <cit.>. We find that imposing this structure greatly improves texture blending between classes. This is likely because the MLP (which acts on these embeddings) can now learn a smoother mapping into the final conditioning signal. As mentioned in note <ref>, this regularization was identified post-publication; hence, our primary results do not make use of this change. We demonstrate its effectiveness in Figure <ref> specifically.



 §.§ Enhancing localized conditioning

Ideally, locally modifying the conditioning feature grid, F, within a confined region should correspondingly alter the generated noise texture solely within that region. However, our empirical observations revealed that such localized adjustments in the conditioning signal often lead to global (or near-global) changes in the resultant noise textures. This phenomenon can be primarily attributed to the architectural design of the employed U-Net neural network – in particular, its bottleneck shape in combination with having many convolutional layers causes the receptive field of each output pixel to be relatively wide. This enables the conditioning signal to spatially propagate across the majority of the canvas, which is undesirable for the quality and controllability of our synthesized noise textures.

To address this, we incorporate a modified version of CutMix data augmentation <cit.> into our training procedure.
CutMix, a technique that is used to enhance performance in image classification tasks, involves creating synthetic training examples by stitching together axis-aligned crops of different dataset samples. Notably, the application of CutMix has been predominantly in classification tasks, with its use in generative models being relatively unexplored and confined to augmenting the performance of GAN discriminators <cit.>. In our context, we apply CutMix by combining noise textures and their corresponding conditioning feature grids, as illustrated in Figure <ref>. This approach not only enriches the diversity of our training dataset, but also imparts a crucial capability to our model: the ability to respond correctly to spatially localized conditioning signals.

We train our network with CutMix data augmentation applied with a probability of 0.5, i.e. for half of the training samples we train without applying CutMix. When augmenting a training image, we sample one to four (uniformly at random) auxiliary noise textures from the dataset, which are then randomly cropped (cut) into rectangular patches with a randomly sampled rotation θ∼𝒰(0,2π). Note that only the crop mask is rotated, not the texture itself. The resulting mixed sample is then a composition of a base noise texture, and one to four texture patches. It is important that all sampled patches belong to unique noise types, otherwise the resulting training image would be invalid, that is, the image would no longer belong to the distribution that we are trying to capture.






§ IMPLEMENTATION DETAILS



  
Noise dataset
We procure a dataset of ∼1.2 million noise images, covering 18 unique noise functions, along with dense samplings of their respective parameter sets (listed in Appendix Table <ref>). In particular, we sample the following noise functions from Adobe Substance 3D Designer: . A preview of our dataset is shown in Figure <ref>. For each noise type, we deterministically sample 16384 parameter sets using the low-discrepancy Halton number sequence, ensuring better coverage over the space of parameters. Each parameter set is sub-sampled four times, i.e. we query the noise functions with four different seeds for each parameter set, resulting in 65536 image samples per noise type, and 1179648 samples across the entire dataset. We sample at a resolution of 512×512px.



  
Network architecture
Our U-Net model is composed of three spatial levels in the encoder and decoder, each level (including the bottleneck) consists of two ResNet blocks that are conditioned on the diffusion timestep, t, and the block's respective SPADE module. The two conditioning signals are composed together as

    γ_2(Z) ⊙(γ_1(t) ·GroupNorm(h) + β_1(t)) + β_2(Z)

where γ_1,β_1 operate on t and γ_2,β_2 belong to a SPADE block. Note that the former functions produce scalar quantities, whereas the latter functions produce spatial modulation maps. Time values are first encoded using sinusoidal positional encoding. The noise parameter MLP is defined by three layers of size 128. The parameter vector f_p is a concatenation of all noise parameters present in the dataset, with entries zeroed out when not applicable. In total our U-Net has ∼5.1 million parameters.



  
Training
Following Song et al. song2020ddim we formulate the U-Net as a noise predictor ϵ_θ(x_t, t, f_c, f_p) making the training objective

    ℒ = 𝔼_ϵ∼𝒩(0,1),t∼𝒰(0,1)ϵ - ϵ_θ(x_t, t, f_c, f_p) _2 + λℒ_reg

We make use of offset noise <cit.>, which replaces the noise sampling, ϵ∼𝒩(0,1), with a slightly modified distribution, 𝒩(0.1δ,1), where δ∼𝒩(0,1). We find that offset noise greatly helps our network resolve noise types that are extremely dark or bright (see Appendix Figure <ref>). We also note that more principled alternatives to offset noise are now available <cit.>. In practice the timestep sampling, t∼𝒰(0,1), is discretized – we make use of a standard number of timesteps, i.e. 1000. We employ the cosine-beta schedule proposed by Nichol et al. nichol2021improvedDiffusion. The spherical embedding regularization loss is weighted by λ=0.02 in Equation <ref>. Training is done in PyTorch; we use the AdamW optimizer <cit.> with a learning rate of 8·10^-5, β_1=0.9, β_2=0.99, and a weight decay of 0.01. We train on 8 NVIDIA RTX 3090 GPUs for ∼300000 optimization steps with batches of size 128. Noise images are downsampled to 256×256px resolution for training.



  
Inference details & performance. Our model can perform 80 diffusion steps per second on a single NVIDIA RTX 3090 GPU at 256×256 resolution at full precision (fp32), scaling quadratically with the resolution. We use the DDIM sampler proposed by Song et al. song2020ddim – unless otherwise specified, our noise figures and visualizations use 30 diffusion steps. To produce tileable noise maps, we modify all  layers in our network to use a circular padding mode, making the canvas topologically toroidal.




§ RESULTS AND EVALUATION



  
Spatially and temporally varying noise
We demonstrate the capabilities of our noise generator by synthesizing noise patterns using an assortment of blending maps, shown in Figures <ref> and <ref>. Our noise generator is able to be conditioned flexibly, allowing one to generate large noise patterns with diverse patterns and blends between them. We additionally show that our model is able to synthesize continuously-varying noise by interpolating the conditioning maps and/or diffusion noise (see Figure <ref> and supp. video). Finally, in Figure <ref> we qualitatively compare our spatially-varying noise to a GAN-based texture synthesis method, PSGAN PeriodicSpatialGAN, as well as a non-parametric texture blending method, ImageMelding ImageMelding.





  
Quantitative evaluation We evaluate our noise generator's spatially-uniform noise textures against the ground truth textures from Adobe Substance 3D Designer using Frechet Inception Distance (FID), which measures the distributional similarity between synthetic samples and the data distribution <cit.>. We sample 20000 synthetic images for each noise type and evaluate the FID of each noise type separately. FID scores are reported in Appendix Table <ref>, alongside analogous scores for PSGAN. Our method achieves a mean FID of 20.9 and median of 13.1, while PSGAN scores 99.2 and 87.5 respectively (lower is better). We note that, for many noise types, PSGAN suffers from mode collapse, which causes virtually all of the synthesized results to have similar visual features and hence its FID is severely impacted. As mentioned in Section <ref>, GAN-based methods struggle to faithfully capture our dataset due to the presence of many disjoint data modes <cit.>. Finally, we show additional results with modified U-Net architectures in Appendix <ref>.



  
Inverse material graph design
We incorporate our noise generator into the differentiable material graph library MATch <cit.>. Given a target photo, we select a template material graph from a selection of 88 graphs using the MATch-provided texture descriptors. In the graph, we replace one noise generator node with our model, and expose optimizable parameters for the class vector f_c, parameter vector f_p, and the diffusion model's latent noise z. We include an L1 regularization on the former vectors to encourage sparsity in the optimized parameters. In Figure <ref> we show that our noise generator provides a valuable prior over the space of noise functions, allowing end-to-end optimization to recover more accurate reconstructions of the target photo. We additionally show results of editing operations on an optimized material graph in Figure <ref>. Target photos are sampled from the Glossy dataset by Zhou et al. zhou2023photomat. Further details for this application are in Appendix <ref>.







  
Tileability and texture size agnosticism
Our model is able to produce noise images that are tileable, which is crucial for many downstream graphics applications. Additionally, we are able to synthesize noise at arbitrary sizes (within computational limits). In tandem, these two properties are highly desired, as it allows one to generate large tileable images that do not contain excessive repeated visual patterns, which is a common problem with tileable images. We  demonstrate these capabilities in Figure <ref>, where we show tileable  and  noise images, as well as the same noises synthesized on canvases that are twice as large, eliminating the repeated visual content. Additionally, in Figure <ref> we show a much larger  pattern that fills a 2048×2048 canvas – we highlight that fine-grained details are not at this size.




  
CutMix augmentation
We demonstrate the affect of our modified CutMix data augmentation strategy by training models with: no augmentation, augmentation with only one patch (), and augmentation with one to four patches sampled uniformly at random (CutMix n∈[1,4]). In Figure <ref>, we see that removing the augmentation strategy severely hinders our model's ability to respond to the conditioning signal in a local manner, producing noise patterns whose characteristics are not well-pronounced. Intuitively, this is because the conditioning signal becomes “blurred” as it propagates through the network's layers. Due to the lack of meaningful ground-truth data, we qualitatively observe that  and  produce similar results, with the latter being able to interpolate noise characteristics more smoothly in some cases.





§ CONCLUSION




We presented a new method for learning a single generative model for a wide range of procedural noise types.
The continuous space of noise patterns learned by our model allows for smooth blending between noise types; the characteristics of the output noise are still controllable by interpretable parameters provided as input to the model.
In addition, our novel approach for spatially-varying conditioning allows the model to generate spatially-varying blends between different noises despite never having seen such data at training time.
We used this model to produce a variety of visually compelling textures, and we showed a proof-of-concept application of how it can be used to improve inverse procedural material design.

Our model is not without limitations.
Not all pairs of noise types can be interpolated well: if the geometric features of two noise patterns are too dissimilar, the intermediate regions of a blend between them can look blurry or otherwise visually awkward (see Fig. <ref>).
It may be possible to reduce these artefacts with further improvements to our CutMix-based data augmentation (e.g. non-rectangular patches; adaptive patch sampling to focus more training time on “difficult” transitions).
But to some extent, such behavior is inevitable, and finding good noise pairs for interpolation is an artistic decision.
It may be possible to leverage the structure of our noise embedding space to suggest good candidates for blending. Finally, diffusion models are known to model low-density modes of the data distribution less accurately <cit.>; in Appendix <ref> we demonstrate our model on additional noise types, and observe that some low-density modes of these noise functions are poorly captured. More careful data sampling techniques may be needed when the desired noise distribution exhibits such modes; e.g. by sampling some regions of the parameter space more frequently.

Procedural material authoring tools such as Adobe Substance Designer often have deterministic pattern generators in addition to stochastic noise generators.
Could our model include these in its learned space of blendable output?
The deterministic nature of these patterns makes them an unnatural fit for generation via denoising diffusion model; modifications to our model and/or training procedure would be necessary to achieve this goal.

It would also be interesting to explore more techniques for interactively creating spatially-vary noise textures.
For example, one might consider an interaction metaphor where the user places `droplets' of noise on a canvas, and those droplets spread an interact with one another by e.g. solving a diffusion equation.

Finally, the application to inverse procedural material design that we presented is an early proof-of-concept; much more could be done in this direction.
Distilling our model into a one-shot diffusion model <cit.> would make it tractable to replace every noise generator node in a procedural material graph with our model.
Provided other graph operations are differentiable, this would open up the possibility of solving for the structure of a material graph via continuous optimization: initializing the graph to be over-complete, assigning weights to edges, and imposing a sparsity prior on edge weights.
Similar approaches have been successfully employed for inferring procedural representations of 3D shapes <cit.>.
In addition, allowing the optimization of spatially-varying noise generators within material graphs might allow for recovering smaller, easier-to-use graphs while still retaining the benefits of interpretable parameters.




This material is based upon work that was supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2040433. Part of this work was done while Arman Maesumi was an intern at Adobe Research.


ACM-Reference-Format







§ EXPERIMENTAL AND IMPLEMENTATION DETAILS






 §.§ Noise dataset details

We include a table enumerating all sampled noise types, the parameters that we sample, as well as the parameter ranges (see Table <ref>). In general, we choose to include any parameters that lead to noticeable changes in the resulting noise, with exceptions to parameters that simply act as color correction (i.e. 's contrast parameter). We note that, despite their names, some of these noises correspond to those that belong to graphics literature; for instance, , , and  are variants of Worley noise <cit.>. For more details about these noises, please refer to the Adobe Substance documentation substanceDocumentation. 

The conditioning vector f_p contains an entry for each unique noise parameter – we treat identical parameter names as separate, with the exception of , which is treated as a single entry in the vector. Parameters are independently normalized to the range [0,1] before being passed to our conditioning MLP. Below we show many samples of our model's outputs with all listed noise parameters being sampled randomly (see <ref>).





 §.§ PSGAN Baseline

We slightly modify the PSGAN PeriodicSpatialGAN architecture for the results shown in Figure <ref>. Since PSGAN is a purely unconditional generative model (i.e. it has no class or parameter conditioning beyond randomized latent code inputs), we endow the PSGAN model with SPADE conditioning blocks that are identical to the ones used throughout our method. We made the discriminator slightly larger to compensate for the added parameters in the generator. Finally, we use the WGAN loss <cit.>. The remaining parts of the network and training are largely kept as-is from the open implementation. In total the generator has 30 million parameters.



 §.§ Inverse Material Design Details

We expose three parameters to the MATch differentiable material graph optimizer; a soft-class vector v_c, the parameter vector f_p, and the diffusion Gaussian noise image z. The soft-class vector represents a list of [0,1] values that are used to index into our class embeddings – this amounts to taking a convex combination over the class embeddings. During optimization, we apply the SoftMax function to v_c to ensure the values are valid. In order to encourage sparsity, we include an L1 regularization term on both v_c and f_p, using a weight of 0.1 for both L1 terms in the final optimization objective. Additionally, we include a scheduled temperature parameter τ into the SoftMax operator by performing element-wise division of v_c by τ, softmax(v, t) = e^v_i/t/∑_j=1^K e^v_j/t.
The temperature is initialized to 0.25 and is updated at every optimization step via the update rule τ' ←τ· 0.97. Finally we clamp τ to be at minimum 0.01. This scheduled temperature forces the optimization to “hone in” on a single class. We use a learning rate of 0.01 for all graphs. After τ = 0.01 we perform a warm-restart of the optimizer and add noise to the exposed parameters to avoid local minima.



§ TRAINING WITH ADDITIONAL NOISE TYPES
 We train our model on two additional noise types that are commonly used in graphics literature: Phasor noise PhasorNoise and Gabor noise lagae2009gabor. We utilize the same data sampling method as mentioned in Section <ref>. The parameters that we sampled for both noise types, as well as their ranges, are enumerated in Table <ref>. We use the released implementations for both noise types to accrue our training data. Examples of spatially-varying images using parameter interpolation and class interpolation for both Phasor and Gabor noise are shown in <ref>.

We note that our model exhibits an FID of 93.6 and 129.3 on Phasor and Gabor noise respectively, which is notably higher than the FID scores for other noise types. Upon inspection of our model's outputs, we see that some parameter configurations are not well captured by our training. For example, Gabor noise exhibits a significantly different visual appearance when its principal frequency and kernel width are very small; however, since this appearance is only captured by a narrow subset of the parameter space, the dataset thus contains much fewer of such samples. One drawback of diffusion models is that they model low-density regions of the data distribution less accurately (compared to higher-density regions)  <cit.>, and hence our performance is worse in such situations. More careful data sampling methods may be needed when the desired noise distribution exhibits low-density modes; e.g. by sampling some regions of the parameter space more frequently.









§ MODEL ARCHITECTURES

The U-Net model detailed in Section <ref> has just ∼5.1 million parameters and is constructed using two downsampling blocks, a bottleneck block, and two upsampling blocks. Each block contains two ResNet sub-blocks. An additional ResNet sub-block is placed at the end of the network. We use channel dimensions of 32 and 64 for the outer blocks, and 128 for the bottleneck. The ResNet sub-blocks are globally conditioned on the diffusion time t, and spatially conditioned (via SPADE) by 128-dim noise embeddings. We will refer to this model as  (extra small) below.

We train two additional model architectures to evaluate the potential performance of larger and more complex models.  is identical to , with the exception of added linear attention layers inside of each block <cit.>, and the addition of an extra set of downsampling/upsampling blocks. Similarly,  is identical to , with the exception of larger channel dimensions. In Table <ref> we summarize these architectures as well as their FID scores and inference performance. The models were trained for the same number of optimization steps following the details in Section <ref>.

For our inverse material design application, we found that using  was most suitable due to the added memory cost of attention layers. For consistency, we used this model for all figures in the main text; however, in applications that do not require such light-weight networks, the larger models may be suitable.
















