A Constructive Prediction of the Generalization Error Across Scales
Jonathan S. Rosenfeld Amir Rosenfeld Yonatan Belinkov Nir Shavit


The spin-forbidden vacuum-ultraviolet absorption spectrum of ^14N^15N
    W. Ubachs
    December 30, 2023
=====================================================================





The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

























§ INTRODUCTION



With the success and heightened adoption of neural networks for real world tasks, some questions remain poorly answered. For a given task and model architecture, how much data would one require to reach a prescribed performance level? 
How big a model would be needed? 

Addressing such questions is made especially difficult by the mounting evidence that large, deep neural networks trained on large-scale data outperform their smaller counterparts, rendering the training of high performance models prohibitively costly.
Indeed, in the absence of practical answers to the above questions, surrogate approaches have proven useful. One such common approach is model scaling, where one designs and compares small-scale models, and applies the obtained architectural principles at a larger scale <cit.>. Despite these heuristics being widely used to various degrees of success, the relation between the performance of a model in the small- and large-scale settings is not well understood. Hence, exploring the limitations or improving the efficiency of such methods remains subject to trial and error.

In this work we circle back to the fundamental question:  what is the (functional) relation between generalization error and model and dataset sizes? Critically, we capitalize on the concept of model scaling in its strictest form: we consider the case where there is some given scaling policy 
that completely defines how to scale up a model from small to large scales. We include in this context all model parameters, such that traversing from one scale (in which all parameters are known) to another requires no additional resources for specifying the model (e.g., architecture search/design). 

We empirically explore the behavior of the generalization error over a wide range of datasets and models in vision and language tasks. While the error landscape seems fairly complex at first glance,
we observe the emergence of several key characteristics shared across benchmarks and domains. Chief among these characteristics is the emergence of regions where power-law behavior approximates the error well both with respect to data size, when holding model size fixed, and vice versa.

 
Motivated by these observations, we establish criteria which a function approximating the error landscape  should meet. 
We propose an intuitive candidate for such a function and evaluate its quality, both in explaining the observed error landscapes and in extrapolating from small scale (seen) to large scale (unseen) errors. 
Critically, our functional approximation of the error depends on both model and data sizes. 
We find that this function leads to a high quality fit and extrapolation. For instance, the mean and standard deviation of the relative errors are under 2% when fitting across all scales investigated and under 5%  when extrapolating from a slimmed-down model (1/16 of the parameters) on a fraction of the training data (1/8 of the examples) on the ImageNet <cit.> and WikiText-103 <cit.> datasets, with similar results for other datasets. 

To the best of our knowledge, this is the first work that provides simultaneously:
 
    
  * A joint functional form of the generalization error landscape—as dependent on both data and model size—with few, interpretable degrees of freedom (section <ref>).
    
  * Direct and complete specification (via the scaling policy) 
    of the model configuration 
    attaining said generalization error across model and dataset sizes.
    
  * Highly accurate approximation of error measurements 
    across model and data scales via the functional form, evaluated on different models, datasets, and tasks (section <ref> ). 
    
    
  * Highly accurate error prediction from small  to large model and data (section <ref>).


We conclude with a discussion of some implications of our findings as a practical and principled tool for  understanding  network design at small scale and for efficient computation and trade-off design in general.  We hope this work also provides a useful empirical leg to stand on and an invitation to search for a theory of generalization error which accounts for our findings.





§ RELATED WORK





  
Model scaling:
A number of studies have explored the effect of model scaling on performance. 
For instance, image classification networks can be scaled by depth <cit.> or width <cit.>. 
More recently, 
<cit.> 
demonstrated how scaling width, depth, and input resolution has combined positive effects larger than  scaling each factor in isolation. However, this relationship has yet to be quantified in a predictive form – by how much will error change with model scaling?

In this work, we focus on finding a constructive functional form for determining the model given a specified performance. 




  
Data scaling:
It has long been recognized that more data improves performance, and various studies report such trends in both computer vision <cit.> and language processing tasks <cit.>. 
A number of prior studies  observed power-law relations between the generalization error and training data size <cit.>. 
Most relevant to our work, <cit.>  explored the effect of data size on the generalization error in vision, language, and speech  tasks, and observed a strikingly consistent power-law behavior in a large set of experiments. However, while these studies point to the empirical existence of a power law in terms of data, they do not offer tools for  predicting the performance given a specified model.
Nor do they offer low-cost methods to specify the model configuration which would attain the power law with data dependency.
Indeed,  had to search over models and their configurations at large scale to exhibit their findings, incurring prohibitive computational costs.

In contrast, we demonstrate a constructive recipe, where we directly predict the test performance at large scale and specify the full model configuration which attains it (with no need for large-scale search), given performance at small scale. 






  
Predicting model performance:
Since training models at full data/model scale may be computationally prohibitive, a line of work tries to predict the performance of a given model  on a given dataset, without training the model, for example by using a bank  of previously trained models, dataset, and their associated performances <cit.>.
Others have proposed to estimate performance on small data <cit.> or model sizes <cit.> in the context of neural architecture search (NAS). In this case, the small-scale evaluation is used to compare models at small cost, to expedite the search process; see <cit.> for a recent survey. 
Our work complements previous approaches by demonstrating a functional form that can predict large-scale performance from small-scale measurements. Moreover, our method may be integrated in NAS, addressing some of its current limitations (as discussed in section <ref>). 




  
Theoretical error bounds:
Much attention has been given to theoretical explanations of the generalization capabilities of deep neural networks <cit.>. While fully engaging with this literature is beyond our scope, we note that 
recent studies have derived bounds involving power-law dependencies in both model <cit.> and data size <cit.>. We leave it as an open question for future work to find theoretical explanations for the empirical behavior and the functional form we investigate in this work. 









§ EXPERIMENTAL SETUP





  
Notation: Let _n = {_i,y_i }_i=1^n denote a labeled (training) dataset with n samples or datapoints.
Let f_m denote a neural network whose size is the number of parameters m, such that ŷ = f_m() is the predicted label.
Let ϵ(n,m  ) be the generalization error as a function of n and m, measured by a performance metric (e.g., top-1 accuracy or cross-entropy loss) on a held-out test set. We refer to this error function as the error landscape. 




 §.§ Scaling Policies





  
Dataset scaling:
We wish to scale datasets while preserving the original distribution.
For image classification, we uniformly subsample all classes by a constant ratio, thus  preserving the relative sample size per class. We limit the maximal sub-sampling to avoid eradicating any class.
For language modeling, where the number of classes (vocabulary items) has a very long tail distribution, we randomly sample sentences such that the total number of sampled words will be a certain fraction of the original dataset. 
Table <ref> reports the data scales we use. 
In all tasks the held-out test set remains untouched for evaluating the error.




  
Model scaling: We are critically interested in a method where moving across scales is defined by some scaling function, such that no additional significant computation would be incurred. We thus  consider the case where the model architecture is given and the model size determines how to scale it. 
For instance, one may scale width (number of channels in convolutional networks, hidden state size in recurrent networks), depth (number of layers),  do compound scaling <cit.>,  or more generally define a function tying the model degrees of freedom and size. 
We focus primarily on width scaling in our experiments; the model scales are reported in Table <ref>. We also perform selected depth scaling to demonstrate flexibility with respect to the scaling method. 




  
Hyper-parameters:
For similar reasons we wish to avoid hyper-paramater search at large scales, and thus 
avoid the temptation to tune hyper-parameters accordingly (learning rate, regularization, etc.). Therefore, we hold all hyper-parameters fixed. 
This 
enables us to construct a functional form that fits the error landscape and can be used to predict the error across scales while completely defining the model attaining it.   We consider pros and cons of this approach in the discussion (section <ref>). 




 §.§ Tasks, Models, Optimizers and Datasets



We experiment with both vision and language tasks.  We use 6  benchmark datasets for image classification and 3 for language modeling. For image classification, we train ResNet <cit.> and WRN models <cit.> with stochastic gradient decent (SGD). In section <ref> we explore the effect of varying architectures and optimizers for a fixed task (CIFAR100), adding VGG16 <cit.> and DenseNet <cit.> models trained with both Adam <cit.> and SGD. 
For language modeling, we train AWD-LSTM <cit.> and Transformer-XL models <cit.> with SGD and Adam optimizers respectively. Summary statistics are shown in Table <ref>, along with the range of explored scales. Appendix <ref> gives additional information. 




§ OBSERVATIONS ON THE ERROR LANDSCAPE









fig_sub:observe_3d_wiki103fig_sub:observe_3d_cifar10_depth=44 respectively show an example test error landscape for width scaling of Transformer-XL on WikiText-103 and WRN-44-16 on CIFAR10. 

Various additional such landscapes are found in appendix <ref>, showing largely consistent patterns. 

Examining the error landscapes yields the following observations:





    
  * Model Scaling 
    
        
  * For a given dataset size, scaling up the model results in an initial decrease in test error, which then saturates to a level determined by the dataset size.[At some point error increase ensues; this point differs between datasets, see  Appendix <ref> for examples.]
         This behavior has been noted by <cit.> across varied model scaling methods, although they have not engaged with the dependency on dataset size.
        
  * The rate of error decrease with model size appears well approximated by a power-law. 
        
        These two observations together can be summarized as the following relation: 
        
    ϵ(m,n) ≈ b(n)m^-β(n) +c_m(n)

        where b, β, c_m  may depend on the data size n, s.t. as m grows, ϵ→ c_m. 
        Example fits to this form (allowing b, β, c_m to be fit per n) are seen in fig_sub:observe_2d_wiki103 (right) and fig_sub:observe_2d_cifar10 (right). 
    
    
  * Data scaling 
    
        
  * For a given model size, scaling up the dataset results in an initial increase in performance, which then saturates to a level determined by the model size. 
        
  * The rate of error decrease with dataset size appears well approximated by a power-law.
        <cit.> 
        also noted a similar relationship, but did not functionally tie the saturation level to the dataset size. 
        
        These two observations together can be summarized as the following relation: 
        
    ϵ(m,n) ≈ a(m)n^-α(m) +c_n(m)

        where a, α, c_n  may depend on the model size m, s.t. as n grows, ϵ→ c_n. Example fits to this form (allowing a, α, c_n to be fit per m) are seen in fig_sub:observe_2d_wiki103 (left) and fig_sub:observe_2d_cifar10 (left).
    
    
  * Joint properties 
    
    The behavior of the error when scaling model size while holding data size fixed, and vice versa, extends to the entire error landscape in a well-behaved manner, such that the manifold ϵ(m,n) is smooth everywhere as a function of both model and data scales. 










§ FUNCTIONAL APPROXIMATION OF THE GENERALIZATION ERROR
 




 §.§ Criteria





Motivated by the above observations, we now consider a functional approximation for the error landscape.  
In particular, 
let us consider function families meeting the following criteria which augment and restrict our observations:



    
  * As either model or dataset size goes to zero, the expected performance is equivalent to a random-guess error level ϵ_0.[Best guess when m → 0 (ϵ_0n) or n → 0 (ϵ_m0) need not coincide, but can, e.g., in a balanced dataset.] 
    
    
  * For a given dataset size, scaling up the model will result in an initial increase in performance, which will then saturate, taking the form in (<ref>).
    
    
  * For a given model size, scaling up the dataset will result in an initial increase in performance, which will then saturate, taking the form in (<ref>). 
    
    
  * There exists an irreducible error ϵ_∞, intrinsic to the dataset. 
    
    
  * The function must be smooth everywhere and monotonic non-increasing in terms of model and data size (observation <ref>). 
    


While there are many possible function families meeting the above criteria, below we propose a simple function family for our evaluation. We do not claim that this is in fact the true underlying dependency, but rather that it serves as a good approximation of the error landscape—consistent with these criteria.




 §.§ Proposed Function Family



As a first insightful step, consider the implications of satisfying  <ref> and <ref> simultaneously. 
By examining the limiting behavior as m or n grow,  we have:



    As m grows large:    c_m(n)    ≈ a(m)n^-α(m)+c_n(m) 
           As n grows large:    c_n(m)    ≈ b(n)m^-β(n)+c_m(n)


Thus, a consistent form satisfying  <ref> and <ref> simultaneously is:

    ϵ(m,n) ≈ a(m)n^-α(m) + b(n)m^-β(n) + c_∞

where c_∞ is a constant not dependent on either m or n.

Let us now examine the simplified case where a,b,α,β are constant: 



    ϵ̃(m,n) = an^-α + bm^-β + c_∞

where  α≥ 0 and β≥ 0 control the global rate at which error decreases with data and model size, respectively, a>0 and b>0 are a form of unit conversion between data and model sizes and error, and c_∞>0 is the asymptotic lower value attainable.
This function is a special case of (<ref>) and meets criteria <ref> and <ref> by construction. Importantly <ref> and <ref> are also met. 

However, by giving up the dependence of a,b,α,β on m,n, this function does not meet criterion <ref>.

We thus need to  model the transition from the initial random-guess level to the power-law region.

We propose to parameterize the transition using the following envelope (complex) function:

    ϵ̂(m,n) = ϵ_0 ‖ϵ̃(m,n)/ϵ̃(m,n)-iη‖ = ϵ_0 ‖an^-α + bm^-β + c_∞/an^-α + bm^-β + c_∞ -iη‖

where i = √(-1). 
Here the simple pole at η controls the transition point from the initial random-guess level ϵ_0 as (m,n) increase. As (m,n) grow,  ϵ̃→ c_∞ and the final irreducible error ϵ_∞≜ϵ_0c_∞η^-1 is approached.
The random-guess error, ϵ_0, is a known parameter determined by dataset statistics (e.g, (N_classes-1) / N_classes for a balanced  dataset). 
Note that due to our choice of rational envelope, we can  divide by a constant the form in  (<ref>). Without loss of generality, let us choose a=1.





Note that while the forms in equations <ref> and <ref> are well motivated, the approach taken for modeling the transition is solely a convenience one. In fact,   the transition(s) as function of m and n may be captured in the functional forms of a,b,α,β or another envelope mechanism. 
We leave a more refined investigation of the nature of the transitions to future work. 







§ ERROR LANDSCAPE ESTIMATION
 





 

We wish to empirically estimate the quality of the proposed functional parameterization as a fit to the true error landscape. 

Let ϵ̂(n,m ; ) be the parametric function family ((<ref>)) approximating the error landscape  ϵ(n,m ), where = {α,β,b,c_∞,η}.[
For image classification, we set ϵ_0 = (N_classes-1) / N_classes (the balanced dataset case). For language modeling, we estimate ϵ_0 as another parameter, such that = {α,β,b,c_∞,η,ϵ_0} in this case. ]

Define the divergence δ(n,m;) as  the  relative difference between the estimated error ϵ̂(m,n;) and the true error ϵ(m,n):

    δ(n,m;) ≜ϵ̂(m,n;)-ϵ(m,n)/ϵ(m,n)


We fit a least squares regression model to find the best parameters minimizing the divergence.
In this section, we fit the function 
using 10-fold cross-validation across all model/data configurations m , n (see Table <ref>) and evaluate the fit quality.  (In the next section, we perform extrapolation experiments, from seen to unseen points.) 
We perform the fit separately for each dataset and evaluate its quality by 
the mean μ and standard deviation σ of the 
divergence δ over all points (m,n). See Appendix <ref> for experimental details.

















As fig:fit shows,  estimated test accuracy is highly correlated with  actual test accuracy for various datasets, with worst-case values μ<1% and σ<5% . Note that the number of free parameters 
is small (||≤ 6) compared to the number of  points ( model-data configurations),
demonstrating the appropriateness of the proposed function 
for modeling the complex error landscape. 





 §.§ A Probe into Depth Scaling
 




 




Here we verify that our results extend to another canonical scaling policy, namely depth scaling. 
fig:cifar10-depth shows the error landscape with depth scaling on  CIFAR10, exhibiting the same characteristics as width scaling. 
fig:fit-cifar10-widthfig:fit-cifar10-depth show error landscape estimation results for both cases of width and depth scaling,
exhibiting small and comparable fit errors  (confidence intervals <3%). 



Since the difference in approximation quality is effectively indistinguishable  when scaling   depth or width orthogonally, we expect compound scaling to adhere to the same functional form. Indeed, we verified this on the publicly available (model scaling only) results for EfficientNet <cit.>.







 §.§ On the Variety  of Optimizers and Architectures

















Our study covers a deliberate variety of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants), following standard implementations in the literature as recommended for each dataset/model setting; 
see Appendix <ref>. 
r0.46 


  
    < g r a p h i c s >

    CIFAR100 Error estimation results with three architectures (WRN, VGG,  DenseNet) and two optimizers (SGD, Adam). 
     
    
    
    
  


 

However, the model/optimizer settings differ in multiple aspects across the different tasks
, rendering the comparison of, say, different optimizers, challenging.

In this section we verify that the functional form holds when varying the optimizer and/or the architecture on the same task, namely image classification on CIFAR100.



In addition to the previously examined setting of WRN with SGD, we add four more settings: two
 well known architectures (VGG and DenseNet), each trained with both SGD and Adam optimizers. See Appendix <ref> for  experimental details. 
 
Figure <ref> exhibits consistent, accurate, fit values across all architecture/optimizer settings, with mean divergence of μ <1% (std: σ <
 6%; confidence intervals <4%).
 
 







 


§ EXTRAPOLATION
 

 

In this section, we evaluate the ability of our functional approximation to extrapolate beyond seen model/data configurations. The primary question we ask is: can we predict the error of a large model/data configuration from the errors of smaller-scale model/data configurations? To do this, we fit the least squares regression on a subset of the configurations and predict the error on larger, unseen configurations. 
More formally, let (m_i, n_j) denote a given model/data configuration. We first estimate parameters _ij by fitting the function in (<ref>) on all points of at most that size (m ≤ m_i, n ≤ n_j).  
Then we predict the  error ϵ(m,n) in all points corresponding to larger configurations (m > m_i, n > n_j) using estimated _ij.  
Finally, we measure the divergence δ(m,n) between the estimated  error and the 
actual  error at all larger configurations. 
This process is illustrated in fig:extrapolation-array.





fig:extrapolation-single-vision shows the results of one such extrapolation experiment, on ImageNet. In this case, we have fit the functional form on all configurations of model size m ≤ m_i = M/16 and data size n ≤ n_j = N/8, and predicted the error on all larger configurations. 
As the figure shows, the extrapolation is highly accurate, with a mean divergence of μ=4.5% (std: σ=4.7%). fig:extrapolation-single-language reports a similar experiment on WikiText-103. Here, again, we see very good extrapolation, with a mean divergence of μ=0.5% (std: σ=1.7%).
Note that each extrapolation is run 10 times with different random initializations of _ij in the least squares with negligible effect on the prediction. 

In practice, we may be interested in  extrapolation quality with different subsets of configurations. Appendix <ref> provides detailed extrapolation results on multiple subsets of configurations, for both vision and language datasets. Generally,  the extrapolation performs well once not ill-posed, which may be caused by lack of signal in the region of the initial “random-guess” level, or in degenerate  cases like having fewer measurements than the number of free parameters in . 







§ DISCUSSION AND CONCLUSION
 


In this work, through insights gained by the joint examination of the dependencies of generalization error on both model and data size, we arrive at criteria for functions consistent with the form of the generalization error under a given scaling policy. We consider one such function and find it to be in very good agreement with the actual behavior of the error landscape. Indeed, the agreement is strong enough  that extrapolation 
from small to large scale  becomes feasible: the function  predicts the behavior of the generalization error in practice for the practical case of scaling models and  data.

We discuss several example implications of knowing such a functional form. 




  
Small-scale network development: 
    At the core of small fidelity searches is the notion of performance rank comparison between models. However, small scale and large scale ranks are not assured to be consistent.
    If indeed a functional form such as empirically found in this work 
    holds very generally, then in contrast, one can safely assess scaling rank between models at small scale, with the assurance that it remains consistent.
    This suggests that one would be well served by searching over scaling policies; a pertinent example of such a success is <cit.>. The functional form also explains the limitation of small-scale search: once reaching the random-guess  error level, where the sensitivity to scaling vanishes,  the informativeness of ranking diminishes. Finally, the functional form allows direct usage of differentiable methods for NAS.




  
Principled design:
Knowing the error landscape function facilitates reasoning about the choice of (m,n) attaining a specified error level. In other words, for any given error level, one can solve Eq. <ref> for m,n based on small-scale measurements. Thus, one can quantitatively answer design questions regarding the expected (in particular, large-scale) relations between m, n, and ϵ.    

 In fact, Eq. <ref> provides direct ansewrs to questions such as "how much data would one require to reach a prescribed performance level?" or "how big a model would be needed?" 
Imposing constraints is also straightforward. For instance, consider the following question: "What is the maximal model size possibly needed (useful), when the data is limited in size, n=n_lim (for a given model architecture and scaling policy)?"


For a fixed dataset size, model scaling eventually contributes marginally to error reduction and becomes negligible when bm^-β≪ n_lim^-α (Eq. <ref>).
Define the relative contribution threshold T as satisfying T = n_lim^-α/ bm_max^-β. (For example, T=10.) Then the maximal useful model size meeting threshold T is:

    m_max(T) = (bT)^1/β n_lim^α/β



Similarly, The maximal useful amount of data for a limited sized model m_lim is:

    n_max(T) = (1/bT)^1/α m_lim^β/α




 Moreover, Eq. <ref> allows for complex design trade-offs. Generally, given some design-tradeoff cost function C(m,n,ϵ), one can minimize such cost s.t. Eq. <ref>. 
For example, consider the case of optimizing for efficient  computation which has both practical and environmental importance <cit.>. 
Since the number of FLOPs during training is ∝ m· n (for constant epoch budget), the trade-off cost function may be formulated as C(FLOPS,ϵ) = C(mn,ϵ). Further, since constant error contour is very well approximated by c = 1/n^α+b/m^β (Eq. <ref>), dataset and models may be scaled  with optimal resource efficiency with no effect on performance by solving for:

    _m,n     m· n         s.t.    c=1/n^α+b/m^β


The solution gives us the optimal-computational-efficiency ratio of model to data size: bβ/αn^α/m^β = 1.




  
Limitations:
    We have made a few simplifying assumptions in our choice of approximating function, in particular in how to model the transition from the initial random-guess  error level and the union of the random-guess level of the two scenarios (small model with large data  and large model with small data). We leave a more detailed examination of the behavior of the transitions from random-guess error levels and refinements of the functional form to  future work.


    Critically, the restrictive nature of our scaling framework (all parameters and hyperparameters described by a policy) is both a blessing and a challenge. The blessing comes in fulfilling the goal of finding simultaneously both the form of the generalization error and the full specification of the model and hyperparameters that attain it across scales. The challenge is that we have demonstrated in this work only the case of constant hyper-parameters. We conjecture that the relation between model configuration and hyperparameter choice <cit.> may entail the potential to formulate hyperparameter-scaling policies similar in nature to the model-scaling polices, and that these too fall under the scope of the form we find in this work. This too will be the subject of future work.
    


We  hope  that this work will bring the actual functional form of the generalization error in this practical case of scaling to the fore, both in practice and as an empirical leg to stand on in the quest for its theoretical origins. 








  §.§.§ Acknowledgments

We thank Alexander Rakhlin, Alexander Madry, Kai Xiao, Lu Mi, Viaks Garg, Dan Alistrah, and Tommi Jaakkola for discussions and their help. We also thank the anonymous reviewers for their valuable feedback.
J.R. was partly supported by the Eli and Dorothy Berman Fellowship as well as grants NSF IIS-1447786, NSF CCF-1563880 and China-Singapore Suzhou Industrial Park.  A.R. was partially supported by the Air Force Office of Scientific Research USA (FA9550-18-1-0054) though a grant to John K. Tsotsos.  Y.B. was partly supported by the Harvard Mind ,Brain, and Behavior Initiative.


iclr2020_conference



§ DATASETS AND MODELS
 




 §.§ Image Classification



  §.§.§ Datasets

We evaluated our predictions on several popular image classification datasets: 
ImageNet <cit.>: a large-scale recognition benchmark consisting of natural images of 1000 object categories with 1.28M training images spread roughly uniformly over the categories. It has 50K validation and 100K testing images. It has been the most popular large-scale benchmark for image classification methods for the better part of the last decade. 
CIFAR10/100 <cit.>: 60K natural RGB images of 10 classes (100 for CIFAR100) with a train/test split of 50K/10K. 
For each of the following datasets, we use the version collated, resized, and split into train/validation/test sets by <cit.>. 
DTD <cit.>: a texture database of 47 categories and 5640 images. Aircraft <cit.>: 10K images of 100 different aircraft classes. UCF101 <cit.>: originally a video action recognition dataset, converted using the method of <cit.> into a single image per video. It contains 13,320 images of 101 action classes.



  §.§.§ Models


We experiment with four models for image classification. We use different variants of the popular ResNet architecture <cit.> in the main experiments. 
For ImageNet we use ResNet-50 and build on the code from the PyTorch framework <cit.> to vary the model width. For all other datasets we use  WRN-44-16 <cit.> of varying widths, modified from the implementation of <cit.>. 






Scaling the models' width is performed by multiplying the number of channels in each convolutional layer and the width of the hidden linear layers by a constant factor and rounding to the nearest integer.
The ranges of width scales (and data scales) for the main experiments are detailed in Table <ref>. 



In section <ref>, we perform width scaling for two additional architectures, VGG16bn <cit.> and DenseNet (L=40, k=32) <cit.>. The VGG and DenseNet models were also modified for width scaling from the implementation of <cit.>. The model scales in this case are  4^-k, 0 ≤ k ≤ 5, for both VGG and DenseNEt.


Depth-scaling, in the CIFAR10 case (section <ref>), is performed by appending extra layers within each block.









































  §.§.§ Training

In the main experiments, training is done via SGD with a momentum of 0.9, weight decay of 1e-4 and initial learning rate of 0.1.
For ImageNet we train for 90 epochs, decreasing the learning rate by a multiplicative factor of 0.1 after and 30 and after 60 epochs. We use a batch size of 16.
For all other vision datasets we use a batch-size of 128. We begin training with a learning rate of 0.1, run for 200 epochs, and reduce by a multiplicative factor of 0.1 after 80, 120, and 160 epochs. 

For the VGG and DenseNet experiments on CIFAR100 in section <ref>, we train with both SGD and Adam optimizers. 
We train VGG for 170 epochs and Densenet for 300 epochs.
Adam hyperparameters are default, with an initial learning rate of 1e-3.
When training with SGD, we retain initial learning rate, batch size, momentum, and weight-decay, as in the main experiment (at 0.1, 128, 0.9, and 1e-4 respectively) and follow standard stepped learning rate schedules: 
For VGG, learning rate multiplicative factor of 0.1 after 80, 120, and 160 epochs; For 
DenseNet, learning rate multiplicative factor of 0.1 after 150 and 225 epochs.








 §.§ Language Modeling




  §.§.§ Datasets

We evaluate on several datasets commonly used for (word-level) language modeling: Penn Treebank <cit.>, WikiText-2 <cit.>, and WikiText-103 <cit.>. 
The PTB is a relatively small language modeling dataset of news texts, with a vocabulary of 10K unique words and about 900K/70K/80K training/validation/test words. 
WikiText-2 is drawn from Wikipedia articles and it is both larger and richer, with a vocabulary of 33K words and 2M/210K/240K training/validation/test words. 
WikiText-103 is also based on Wikipedia, but larger still, with a vocabulary of 270K words and 100M training words (and the same validation and test sets as WikiText-2).  






  §.§.§ Models

We experiment with two standard models for language modeling: Transformer-XL <cit.> and AWD-LSTM <cit.>. 
Transformer-XL is a recent language modeling architecture that is based on transformer self-attention <cit.>, but modified to better learn dependencies beyond a fixed length by adding a segment-level recurrence mechanism. It has achieved state-of-the-art results on multiple benchmarks. We use the official PyTorch implementation[<https://github.com/kimiyoung/transformer-xl>] with their base configuration: 16 layers, embedding size of 410, inner dimension of 2100 in the fully-connected layers, and 10 attention heads. Training is done with Adam. See the implementation for other details. For scaling experiments, we decimate the inner dimension. We use Transformer-XL for WikiText-103. 

AWD-LSTM is a long short-term memory <cit.> language model with adaptive weight averaging. We use the  official implementation[<https://github.com/salesforce/awd-lstm-lm>]  with the recommended configuration:
3 layers, embedding size of 400, and hidden state size of 1150. Training is done with SGD. We use AWD-LSTM for PTB and WikiText-2 and follow the recommended settings for these two datasets.  
For scaling experiments, we decimate the hidden state size. 







§ ACCESSING INSIGHTS GIVEN ERROR LANDSCAPE



 §.§ Error Landscape Regions - Useful Approximations

The approximate form in <ref> by construction mirrors the criteria in <ref> as a special case of eq. <ref>. For this approximation, reviewing its regions can be useful in guiding network development as mentioned in the discussion section. For ease of analysis we highlight here some useful simplified forms for the different error regions and their implications for estimation. 


    
    
  * Random guess initial plateau region.
    In this region either m or n are small such that ϵ̃≫η and the error ϵ∼ϵ_0 is weakly dependent on m,n. We note again explicitly that error when m → 0 (ϵ_0n) or n → 0 (ϵ_m0) need not coincide, but can, e.g., in a balanced dataset.
    In this region one can only (trivially) estimate (measure) the random guess level.
    
  * Saturating power-law region: an^-α + bm^-β≪η (recall a=1 in eq. <ref>). In this region eq. <ref> simplifies to the form of eq. <ref>: 
    
    ϵ̂ =   ϵ_0 /ηϵ̃ = ϵ_0 /η(n^-α + bm^-β + c_∞) = a'n^-α + b'm^-β + ϵ_∞

    
    

    Note that for informative data there is a large difference between the random guess level and the Bayes error, such that there is a large region satisfying simultaneously c_∞≪ an^-α + bm^-β≪η
    Thus upon entering the transition to this region, estimation of  a',b',α,β becomes possible. 
    
    
    
  * Finally, when m,n are large enough such that the effect of c_∞ (equivalently the Bayes error ϵ_∞) is not negligible, one can directly estimate it. 
    Critically, since ϵ_∞ is expected to be a property solely of the dataset, for the sake of comparing models or other scaling policies for a given task, estimating c_∞ is not needed.
     




 §.§ Design Q&A


In this section we review example questions which are answered by accessing the form of the error (eq. <ref>) - with the aim of exemplifying how to address such design questions. 

Consider a set of measurements, at small scales m,n such that we estimate b,α,β,η,ϵ_0 (at small scale the estimation of c_∞ is 0). 

We are interested in the large scale errors, so eq. <ref> holds and simplifies further to:


    ϵ̂ = a'n^-α + b'm^-β


Importantly, only when for very large scales (when the Bayes error is approached) the above estimate becomes over optimistic (effectively estimating ϵ_∞ = 0 instead of its finite value). 


    
  * Given a model/optimization/task - how much data, and how big a model are needed for attaining a certain error level?
    
    For a prescribed error level ϵ_target the immediate implication is that we have a continuous design space (constant error contour) of possible set of m,n satisfying the target error:
    
    ϵ_target =a'n^-α + b'm^-β

    
    Special cases of this contour are the following questions:
    
  * What is the minimal amount of data needed, assuming no limit in possible model size?
    
    The answer is directly given by recognizing that when the model size is 'large' compared to the data a'n^-α≫ b'm^-β, the error is given by 
    ϵ_target =a'n^-α

    and thus:
    
    n_min = (1/η·ϵ_0/ϵ_target)^1/α

    
  * Similarily, the minimal necessary model size, for abundant data:
    
    
    m_min = (b/η·ϵ_0/ϵ_target)^1/β

    
    
  * What is the maximal amount of data possibly needed (useful), when the model is limited in size m=m_lim?
    
    For a fixed model size, data addition contributes marginally and eventually becomes negligible when a'n^-α≪ b'm_lim^-β.
    
    For concreteness let us define the relative contribution threshold T as satisfying 
    
    T = b'm_lim^-β/a'n_max^-α
 
    
    For example, T=10.
    
    Then the maximal useful data meeting threshold T is:
    
    
    n_max(T) = (T/b)^1/α m_lim^β/α

    
    
  * Similarly, What is the maximal useful model size for a limited amount of data (n=n_lim):
    
    
    m_max(T) = (bT)^1/β n_lim^α/β

    




§ DERIVATION OF COMPUTATIONALLY EFFICIENT TRAINING MODEL-DATA RATIO


In the discussion we consider the example tradeoff minimization problem:


    _m,n      m· n   s.t.      c=1/n^α+b/m^β


The solution is readily found by introducing a Lagrange multiplier, arriving at the equivalent problem:

    _m,nℒ

Where:

    ℒ =m· n + λ[c-( 1/n^α+b/m^β)]


Requiring stationary we have:


    (a)      0 = ∂ℒ/∂ m   = n + λβ b m^-β-1
    
    (b)      0 = ∂ℒ/∂ n   = m + λα n^-α-1

From (b) we have:


    (c)      λ   = -m/αn^α+1


plugging (c) into (a) we arrive at the sought solution (easily verified to be a minimum):




    bβ/αn^α/m^β = 1









§ ERROR ESTIMATION EXPERIMENT




 §.§ Experimental Details



In the experiment described in section <ref>, 
we fit a least squares regression model to find the best parameters minimizing the divergence δ(m,n) - evaluated at configurations m,n as in Table 
<ref>:  

    ^* = _∑_n, m | δ(m,n;) |^2

We quantify the quality of the fit by 
the mean μ and standard deviation σ of the 
fitted divergence by performing standard 10-fold cross validation over all points (m,n) with confidence intervals reported as ± 1 std over the folds.




 §.§ Found Theta Values
 


The  attaining the best fit are listed in Table <ref>.
Since these values all have a physical meaning, it is interesting to consider the values. Let us examine some of the implications arising:

    
  * Of special interest are the values of α and β which control the rate at which the generalization error scales with data and model size. We did not constrain in our optimization over  any parameter, apart from requiring that they all be non-negative. Empirically, the optimal α and β take on values which are meaningful fractions and up to ≈ 1.
    Comparing these values to theoretical expectations is beyond the scope of this work, but is likely a fruitful path to pursue in search for consistent theory.
    
  * The value of ϵ_∞, which is directly calculable by definition as ϵ_∞≜ϵ_0c_∞η^-1, provides an interesting prediction:
    for the given dataset's underlying distribution (e.g., no additional priors added such as data-augmentation, no change of the dataset dimensionality such as resolution increase, etc.), ϵ_∞ is the lowest possible error.
    
  * Another interesting quantity which arises in the language modeling case is the (fitted) value of ϵ_0. We recall that in this case it corresponds to the random-guess cross entropy loss level. The interpretation of this quantity is the cross entropy between the model output distribution and the corpus distribution. It is interesting to note that the values attained here are lower than a unigram model which one might consider as good benchmarks for the case of data far exceeding model capacity.
    







.
2







imagenet: [ 0.75575083  0.75403879  0.61131518  3.62934233 18.50376969  ]

cifar10: [5.87019717e-02 6.55043783e-01 5.34102925e-01 7.14208136e-14
 1.97701518e+01 ]
 
decathlon_dtd: [4.30356871e-05 4.00319211e-01 1.16231333e+00 1.26566374e-09
 8.46839835e-01 ]
 
decathlon_aircraft: [3.47121817e-03 1.10233368e+00 8.31731092e-01 5.16320851e-10
 1.12529537e+00 ]
 
decathlon_cifar100: [0.14727227 0.70403326 0.50562759 0.70969734 6.92618391 ]

decathlon_ucf101: [4.67808736e-02 9.33547255e-01 5.37578077e-01 1.15612716e-09
 2.98124532e+00 ]


wiki103, theta: [ 0.32914295  0.73505031  0.55718887  9.03598661 16.33563873  6.59633058]
PTB, theta: [0.14690378 0.80962791 0.34315027 4.99807364 6.27494232 6.09699692]
wiki2, theta: [ 0.99145936  1.00822978  0.21667458  8.23497095 10.37612973  6.21205331]









§ ADDITIONAL ERROR LANDSCAPE MEASUREMENTS AND ESTIMATIONS
 

In this appendix, we provide error landscape measurements and estimations for all datasets, corresponding to the experiment in section <ref>. 
The results are shown in 3D graphs similar to fig:landscape-3d. 
In each such graph, the z-axis is the logarithm of the generalization error as a function of two independent variables: the model size m and the data size n. 









The 3D graph is deliberately portrayed in log-log-log scale, as we cover a very large range of data scales and model scales and a correspondingly wide range of errors. This view is a useful one when one wishes to evaluate both large dynamic ranges (simultaneously both very large and very small values) and is especially vivid in portraying power-law like dependencies; a power-law naturally forms a straight line in a log-log view.









In each figure, subfigure (a) shows  the measured error landscape is in log-log-log scale, where each point (blue dot) is the error resulting from training with a model/data configuration m,n. Subfigure (b) shows the best-fit estimated error landscape. 
The surface is a linear interpolation between the points, which is then projected on the model-error (m,ϵ), data-error (n,ϵ), and model-data (m,n) planes. 
The contour plots on each one of these planes are the projections of the error landscape surface, and are useful in considering the behavior of the surface when holding one dimension constant. 

We call to attention several interesting observations on the datasets explored:

    
  * As quantified rigorously in section <ref>, the fits perform well across error ranges. In these surfaces, one also gets qualitative sense of the fit adequacy across the wide ranges of the dataset and model scales directly. While perhaps slightly difficult to asses the surface directly, a helpful view is to consider the similarity between the projections of the actual and projected surfaces.
    
  * With increasing model size, indeed typically the error does remain saturated. However, in one of our tested datasets (fig:appB_ucf101) there was a renewed slight increase. We verify that this is indeed over-fitting, in the sense that there is no corresponding increase in the training error.
    We note that the functional form we find can actually be used to veer clear of the m,n regions where such over-fitting may occur.
    
  * The simplifying approach taken by considering the random guess levels (and associated transitions) for small models or small data as identical, seems to work fairly well with some deviation apparent by examining fig:appB_wiki103. Indeed the simplification can hold well for balanced datasets, but need not for imbalanced ones such as in the task of language modeling. Thus, a relaxation of this simplification is expected to be important conceptually and practically.
    






















 
















§ ADDITIONAL EXTRAPOLATION RESULTS
 


Here we provide detailed extrapolation results, for all datasets. All figures are structured in a similar way. 

Each subplot shows estimated (y-axis)  vs.  actual error (x-axis) (0 to 1 scale on both axes). Each subplot is located at the coordinate of the maximal data and model given for the task of performing the fit to the functional form in (<ref>). 
This is the point at the top-right corner of the green dots in the illustration in fig:extrapolation-array.
The target is to find the error-landscape values for unseen, larger scales of both model and data (red points in the same illustration). 
Going from left to right in each figure indicates observed measurements of the error from models of an increasing fraction w.r.t the full size. Going from bottom-to top indicates observed measurements of the error from dataset sizes of an increasingly large fraction of the full dataset.

In each subplot, every point shows the estimated vs. actual error on a model-data configuration. Points that were given for fitting the function are colored in green, while unseen points that were not used are in red. 

The red points show  the estimation error vs.  actual error when extrapolating to all larger models and data sizes. 
In each subplot,  the mean and standard deviation over all divergences δ at target points are given in text. 

Each experiment fit of the parameters was repeated 100 times, with different random initializations of . The shaded bands show one standard deviation across these runs. 

The quality of the extrapolation is critically dependent on the signal provided in the (green) fitted points. Two limiting factors are evident by examining the figures below, which both play a role in the well-posedness of the solution:

    
  * The proximity to the initial random guess level. Only upon transitioning from the initial error plateau, does meaningful signal about the scaling rates become available. Indeed, for scales prior still in the region or close to the initial error level, one sees poor extrapolation results; see figures <ref>,   <ref>, and <ref>, and the vivid origin of this phenomena by examining figures <ref>,   <ref>, and <ref>.
    
  * A second source of ill-posedness is tied to the number of configurations used for the estimation of . Clearly, when this is small,  one cannot expect the extrapolation to be stable. In fact, at least two measurements in each scaling dimension (model/data) are needed, and no less than the number of parameters in  in total. Indeed, for all the plots in this appendix, the smallest scale of m,n is omitted form the graph such that the lowermost row and leftmost column span exactly two model and data scales correspondingly. Of course, there is nothing tying directly the number of points and scale of configurations measured, and one can decouple these two factors by taking closer spaced samples at small scale.
    
  * When both the above factors are not limiting the measurement, one readily sees that for divergences of no more than a few percent, it is sufficient to measure model/data configurations which are far-ranged from the configurations which one wishes to extrapolate to .






 
 
  
  
 
 
  
 
 
 
 

  
 
 
 
 
   
 
  

 
 
 

































