<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>

<p dir="auto">automatically inferring the input size after the first forward pass</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto"><a href="https://mxnet.apache.org/api/python/docs/tutorials/getting-started/to-mxnet/pytorch.html" rel="nofollow">https://mxnet.apache.org/api/python/docs/tutorials/getting-started/to-mxnet/pytorch.html</a><br>
In PyTorch you have to specify the input size as the first argument of the Linear object. Apache MXNet provides an extra flexibility to network structure by automatically inferring the input size after the first forward pass.</p>

<h2 dir="auto">Pitch</h2>
<p dir="auto">I have to specify input of the hidden layer input. I want to make sequence without specifing input size.<br>
Exp: #for now;<br>
pt_net = pt_nn.Sequential(<br>
pt_nn.Linear(28*28, 256),<br>
pt_nn.ReLU(),<br>
pt_nn.Linear(256, 10))</p>
<p dir="auto">My request; #input size is know and second layer input size also known<br>
pt_net = pt_nn.Sequential(<br>
pt_nn.Linear(256),<br>
pt_nn.ReLU(),<br>
pt_nn.Linear(10))</p>

<h2 dir="auto">Alternatives</h2>

<h2 dir="auto">Additional context</h2>
<h2 dir="auto">Motivation</h2>
<p dir="auto">There’s a lot of cases in framework design where we want to be able to treat components like lego blocks when building various model architectures. The following case studies in PyText are the 90% common cases, not the exceptions, ie. most people build models from these architecture families.</p>
<p dir="auto">Document classifiers in PyText follow a general pattern of Embedding → Representation → Decoder.</p>
<ul dir="auto">
<li>Embedding
<ul dir="auto">
<li>Word embedding</li>
<li>BiTransformer</li>
<li>Character embedding</li>
</ul>
</li>
<li>Representation
<ul dir="auto">
<li>Encoder
<ul dir="auto">
<li>DocNN</li>
<li>LSTM (possibly bidirectional)</li>
</ul>
</li>
<li>Pooling layer
<ul dir="auto">
<li>Max pool</li>
<li>Attention</li>
<li>etc</li>
</ul>
</li>
</ul>
</li>
<li>Decoder
<ul dir="auto">
<li>Almost always an MLP, needs to have an output dimension of #classes</li>
</ul>
</li>
</ul>
<p dir="auto">Within the 4 categories of Embedding/Encoder/Pooling/Decoder though, each boundary depends on the dimensionality of the previous component — the encoder depends on the embedding dimension, the pooling depends on the encoder dimension (including eg. whether the LSTM is bidirectional), and the decoder depends on the pooling layer output dimension (and #classes).</p>
<p dir="auto">However, from a user standpoint, it is extremely desirable to not have to make sure that these dimensions match up exactly. For instance, if a product team wants to experiment with DocNN vs BiLSTM with self attention and two different sizes of word embeddings, they would need to know how to compute the correct input dimensions for their encoders, pooling layers, and MLPs for each of the 4 cases. If they have dense features, those might also factor into the computation for the MLP layer.</p>
<p dir="auto">PyText solves this problem by having modules of each of these categories compute and track the appropriate dimensionality internally to their constructor, and then constructing the model and wiring the various blocks together before training. See <a href="https://github.com/facebookresearch/pytext/blob/a64370c7f1aac1c0491ccf5b2b0987909c9e0d94/pytext/models/representations/docnn.py#L30">DocNN</a> representation_dim computation for an example of this, and <a href="https://github.com/facebookresearch/pytext/blob/a64370c7f1aac1c0491ccf5b2b0987909c9e0d94/pytext/models/model.py#L167">Model.from_config</a> for where we use this to construct the model.</p>
<p dir="auto">This solution works for us, but only in the case where we take the responsibility of constructing the model, along with all of its subcomponents, away from the user. If we wanted to eg. allow the user to pass in their own representation layer, we wouldn’t be able to ensure the dimensionality match, the user is therefore responsible for also constructing <em>every other model component</em>. As a direct result of this design, PyText is abysmally hard to use in a notebook, and we built a Config system for managing hyperparameters that essentially boils down to allowing users to partially construct modules, then let us take over and finish building the whole model. See one of our <a href="https://github.com/facebookresearch/pytext/blob/master/pytext/models/representations/bilstm_doc_attention.py#L35">BiLSTM Config</a> classes for an example.</p>
<p dir="auto">This issue is not specific to PyText. Consider this snippet from the demo example from the AllenNLP website:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="EMBEDDING_DIM = 6
HIDDEN_DIM = 6

token_embedding = Embedding(num_embeddings=vocab.get_vocab_size('tokens'),
embedding_dim=EMBEDDING_DIM)
word_embeddings = BasicTextFieldEmbedder({&quot;tokens&quot;: token_embedding})
lstm = PytorchSeq2SeqWrapper(torch.nn.LSTM(EMBEDDING_DIM, HIDDEN_DIM, batch_first=True))
model = LstmTagger(word_embeddings, lstm, vocab)"><pre class="notranslate"><span class="pl-v">EMBEDDING_DIM</span> <span class="pl-c1">=</span> <span class="pl-c1">6</span>
<span class="pl-v">HIDDEN_DIM</span> <span class="pl-c1">=</span> <span class="pl-c1">6</span>

<span class="pl-s1">token_embedding</span> <span class="pl-c1">=</span> <span class="pl-v">Embedding</span>(<span class="pl-s1">num_embeddings</span><span class="pl-c1">=</span><span class="pl-s1">vocab</span>.<span class="pl-en">get_vocab_size</span>(<span class="pl-s">'tokens'</span>),
<span class="pl-s1">embedding_dim</span><span class="pl-c1">=</span><span class="pl-v">EMBEDDING_DIM</span>)
<span class="pl-s1">word_embeddings</span> <span class="pl-c1">=</span> <span class="pl-v">BasicTextFieldEmbedder</span>({<span class="pl-s">"tokens"</span>: <span class="pl-s1">token_embedding</span>})
<span class="pl-s1">lstm</span> <span class="pl-c1">=</span> <span class="pl-v">PytorchSeq2SeqWrapper</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">nn</span>.<span class="pl-v">LSTM</span>(<span class="pl-v">EMBEDDING_DIM</span>, <span class="pl-v">HIDDEN_DIM</span>, <span class="pl-s1">batch_first</span><span class="pl-c1">=</span><span class="pl-c1">True</span>))
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">LstmTagger</span>(<span class="pl-s1">word_embeddings</span>, <span class="pl-s1">lstm</span>, <span class="pl-s1">vocab</span>)</pre></div>
<p dir="auto">(source - <a href="https://allennlp.org/tutorials" rel="nofollow">https://allennlp.org/tutorials</a>)</p>
<p dir="auto">Note that similar to PyText, we need to match up the EMBEDDING_DIM between token_embedding and the LSTM module; we therefore couldn’t construct an LstmTagger object with a default argument for just one of word_embeddings or lstm, because there would be no way to create them with the appropriate dimensions.</p>
<h2 dir="auto">Proposal - Lazy Modules</h2>
<p dir="auto">We propose adding lazily inferred dimensions to some core Torch components, such as Linear, ConvNd, and RNN. This syntax should allow constructing these components (and any derived or more complex user components containing them) to instantiate the models without fully defining their dimensions, and have them inferred from the tensors passed in the first time their forward function is called. We propose using -1 to represent these dimensions similar to how -1 is used in reshape now (None has also been suggested, covered in alternatives below). For example</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; l = nn.Linear(-1, 3)
&gt;&gt;&gt; l(torch.rand(2, 4)).size()
[2, 3]
&gt;&gt;&gt; l(torch.rand(2, 5))
TypeError(...)"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">l</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-c1">3</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">l</span>(<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">2</span>, <span class="pl-c1">4</span>)).<span class="pl-en">size</span>()
[<span class="pl-c1">2</span>, <span class="pl-c1">3</span>]
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">l</span>(<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">2</span>, <span class="pl-c1">5</span>))
<span class="pl-v">TypeError</span>(...)</pre></div>
<p dir="auto">This would allow building more complex components that could also infer their input dimensions; consider a simple MLP implementation, much like what PyText uses for its decoders:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class MLP(nn.Module):
    def __init__(self, dims: List[int], dropout: float = 0.):
        super().__init__()
        layers = []
        for in_dim, dim in zip([-1] + dims, dims):
            layers.extend([
                nn.Linear(in_dim, dim),
                ReLU(),
                nn.LayerNorm(dim),
                nn.Dropout(dropout),
            ])
        self.layers = nn.Sequential(*layers)

    def forward(self, input: torch.Tensor) -&gt; torch.Tensor:
        return self.layers(input)"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">MLP</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">dims</span>: <span class="pl-v">List</span>[<span class="pl-s1">int</span>], <span class="pl-s1">dropout</span>: <span class="pl-s1">float</span> <span class="pl-c1">=</span> <span class="pl-c1">0.</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>()
        <span class="pl-s1">layers</span> <span class="pl-c1">=</span> []
        <span class="pl-k">for</span> <span class="pl-s1">in_dim</span>, <span class="pl-s1">dim</span> <span class="pl-c1">in</span> <span class="pl-en">zip</span>([<span class="pl-c1">-</span><span class="pl-c1">1</span>] <span class="pl-c1">+</span> <span class="pl-s1">dims</span>, <span class="pl-s1">dims</span>):
            <span class="pl-s1">layers</span>.<span class="pl-en">extend</span>([
                <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-s1">in_dim</span>, <span class="pl-s1">dim</span>),
                <span class="pl-v">ReLU</span>(),
                <span class="pl-s1">nn</span>.<span class="pl-v">LayerNorm</span>(<span class="pl-s1">dim</span>),
                <span class="pl-s1">nn</span>.<span class="pl-v">Dropout</span>(<span class="pl-s1">dropout</span>),
            ])
        <span class="pl-s1">self</span>.<span class="pl-s1">layers</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Sequential</span>(<span class="pl-c1">*</span><span class="pl-s1">layers</span>)

    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>: <span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>) <span class="pl-c1">-&gt;</span> <span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>:
        <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">layers</span>(<span class="pl-s1">input</span>)</pre></div>
<p dir="auto">Now this MLP can be used in any PyText model, regardless of what the output shape of representation/pooling is:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="mlp1 = MLP([4, 5, 6])
mlp2 = MLP([4, 5, 6])

mlp1(torch.rand(2, 4))  # finalizes mlp1 as expecting input dimension 4
mlp2(torch.rand(2, 5))  # finalizes mlp2 as expecting input dimension 5"><pre class="notranslate"><span class="pl-s1">mlp1</span> <span class="pl-c1">=</span> <span class="pl-v">MLP</span>([<span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">6</span>])
<span class="pl-s1">mlp2</span> <span class="pl-c1">=</span> <span class="pl-v">MLP</span>([<span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">6</span>])

<span class="pl-en">mlp1</span>(<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">2</span>, <span class="pl-c1">4</span>))  <span class="pl-c"># finalizes mlp1 as expecting input dimension 4</span>
<span class="pl-en">mlp2</span>(<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">2</span>, <span class="pl-c1">5</span>))  <span class="pl-c"># finalizes mlp2 as expecting input dimension 5</span></pre></div>
<p dir="auto">And in the cases of PyText and AllenNLP these could allow interfaces that are much simpler and more modular:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class Model(nn.Module):
    def __init__(
        self,
        embedding = WordEmbedding(),
        representation = BiLSTM(),
        decoder = MLP([4, 5, 6]),
    ):
        self.embedding = embedding
        self.representation = representation
        self.decoder = decoder
    
    def forward(self, tokens):
        embedded = self.embedding(tokens)
        representation = self.representation(embedded)
        return self.decoder(representation)
        
model1 = Model()
model2 = Model(representation=DocNN())
model3 = Model(
    embedding=WordEmbedding(embedding_dim=200, vocab_size=10000),
    decoder=MLP([2, 50, 10]),
)"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">Model</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(
        <span class="pl-s1">self</span>,
        <span class="pl-s1">embedding</span> <span class="pl-c1">=</span> <span class="pl-v">WordEmbedding</span>(),
        <span class="pl-s1">representation</span> <span class="pl-c1">=</span> <span class="pl-v">BiLSTM</span>(),
        <span class="pl-s1">decoder</span> <span class="pl-c1">=</span> <span class="pl-v">MLP</span>([<span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">6</span>]),
    ):
        <span class="pl-s1">self</span>.<span class="pl-s1">embedding</span> <span class="pl-c1">=</span> <span class="pl-s1">embedding</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">representation</span> <span class="pl-c1">=</span> <span class="pl-s1">representation</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">decoder</span> <span class="pl-c1">=</span> <span class="pl-s1">decoder</span>
    
    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">tokens</span>):
        <span class="pl-s1">embedded</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">embedding</span>(<span class="pl-s1">tokens</span>)
        <span class="pl-s1">representation</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">representation</span>(<span class="pl-s1">embedded</span>)
        <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">decoder</span>(<span class="pl-s1">representation</span>)
        
<span class="pl-s1">model1</span> <span class="pl-c1">=</span> <span class="pl-v">Model</span>()
<span class="pl-s1">model2</span> <span class="pl-c1">=</span> <span class="pl-v">Model</span>(<span class="pl-s1">representation</span><span class="pl-c1">=</span><span class="pl-v">DocNN</span>())
<span class="pl-s1">model3</span> <span class="pl-c1">=</span> <span class="pl-v">Model</span>(
    <span class="pl-s1">embedding</span><span class="pl-c1">=</span><span class="pl-v">WordEmbedding</span>(<span class="pl-s1">embedding_dim</span><span class="pl-c1">=</span><span class="pl-c1">200</span>, <span class="pl-s1">vocab_size</span><span class="pl-c1">=</span><span class="pl-c1">10000</span>),
    <span class="pl-s1">decoder</span><span class="pl-c1">=</span><span class="pl-v">MLP</span>([<span class="pl-c1">2</span>, <span class="pl-c1">50</span>, <span class="pl-c1">10</span>]),
)</pre></div>
<h2 dir="auto">Sample implementation</h2>
<p dir="auto">This will be supplied shortly following this proposal as a pull request in torch.nn.lazy, but for now here is the proposed implementation.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from torch import nn

# LazyModuleMeta re-implements the type construction semantics for objects to allow
# a slight variant on syntax. Essentially anything with this metaclass can optionally
# execute a single yield statement during its constructor (normal constructors also work fine).
# If it does yield during construction, then a __lazy_init__ function is populated;
# any code occurring before yield in the constuctor will be called as normal during object creation,
# and any code after yield will instead be deferred to the first call of __lazy_init__.

class LazyInitMeta(type):
    def __call__(cls, *args, **kwargs):
        if hasattr(cls, '__new__'):
            obj = cls.__new__(cls, *args, **kwargs)
        else:
            obj = object.__new__(cls)
        
        def initialize(obj):
            res = obj.__init__(*args, **kwargs)
            if isinstance(res, types.GeneratorType):
                next(res, None)
                def lazy_init(call_args):
                    try:
                        res.send(call_args)
                    except StopIteration:
                        pass
                    finally:
                        obj.__lazy_init__ = None
                obj.__lazy_init__ = lazy_init
            else:
                obj.__lazy_init__ = None
            
        if isinstance(obj, cls):
            initialize(obj)
        return obj
        
# Here's a Lazy nn.Module implementation using LazyInitMeta, calling __lazy_init__ before the first
# forward pass.
    
class LazyModule(nn.Module, metaclass=LazyInitMeta):
    def __init__(self):
        nn.Module.__init__(self)
    
    def __call__(self, *args, **kwargs):
        if self.__lazy_init__:
            self.__lazy_init__(call_args=(args, kwargs))
        return nn.Module.__call__(self, *args, **kwargs)
        
# Optionally lazy Linear module, based on the current torch implementation of nn.Linear.

class Linear(nn.Linear, LazyModule):
    def __init__(self, in_features, out_features, bias=True):
        LazyModule.__init__(self)
        self.in_features = in_features
        self.out_features = out_features
        if bias:
            self.bias = nn.Parameter(torch.Tensor(out_features))
        else:
            self.register_parameter('bias', None)
            
        if in_features == -1:
            self.register_parameter('weight', None)
            ([input], _) = yield  # lazy init remainder
            in_features = self.in_features = input.size()[-1]
            
        self.weight = nn.Parameter(torch.Tensor(out_features, in_features))
        self.reset_parameters()"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">torch</span> <span class="pl-k">import</span> <span class="pl-s1">nn</span>

<span class="pl-c"># LazyModuleMeta re-implements the type construction semantics for objects to allow</span>
<span class="pl-c"># a slight variant on syntax. Essentially anything with this metaclass can optionally</span>
<span class="pl-c"># execute a single yield statement during its constructor (normal constructors also work fine).</span>
<span class="pl-c"># If it does yield during construction, then a __lazy_init__ function is populated;</span>
<span class="pl-c"># any code occurring before yield in the constuctor will be called as normal during object creation,</span>
<span class="pl-c"># and any code after yield will instead be deferred to the first call of __lazy_init__.</span>

<span class="pl-k">class</span> <span class="pl-v">LazyInitMeta</span>(<span class="pl-s1">type</span>):
    <span class="pl-k">def</span> <span class="pl-en">__call__</span>(<span class="pl-s1">cls</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-k">if</span> <span class="pl-en">hasattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s">'__new__'</span>):
            <span class="pl-s1">obj</span> <span class="pl-c1">=</span> <span class="pl-s1">cls</span>.<span class="pl-en">__new__</span>(<span class="pl-s1">cls</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-k">else</span>:
            <span class="pl-s1">obj</span> <span class="pl-c1">=</span> <span class="pl-s1">object</span>.<span class="pl-en">__new__</span>(<span class="pl-s1">cls</span>)
        
        <span class="pl-k">def</span> <span class="pl-en">initialize</span>(<span class="pl-s1">obj</span>):
            <span class="pl-s1">res</span> <span class="pl-c1">=</span> <span class="pl-s1">obj</span>.<span class="pl-en">__init__</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
            <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">res</span>, <span class="pl-s1">types</span>.<span class="pl-v">GeneratorType</span>):
                <span class="pl-en">next</span>(<span class="pl-s1">res</span>, <span class="pl-c1">None</span>)
                <span class="pl-k">def</span> <span class="pl-en">lazy_init</span>(<span class="pl-s1">call_args</span>):
                    <span class="pl-k">try</span>:
                        <span class="pl-s1">res</span>.<span class="pl-en">send</span>(<span class="pl-s1">call_args</span>)
                    <span class="pl-k">except</span> <span class="pl-v">StopIteration</span>:
                        <span class="pl-k">pass</span>
                    <span class="pl-k">finally</span>:
                        <span class="pl-s1">obj</span>.<span class="pl-s1">__lazy_init__</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
                <span class="pl-s1">obj</span>.<span class="pl-s1">__lazy_init__</span> <span class="pl-c1">=</span> <span class="pl-s1">lazy_init</span>
            <span class="pl-k">else</span>:
                <span class="pl-s1">obj</span>.<span class="pl-s1">__lazy_init__</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
            
        <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">obj</span>, <span class="pl-s1">cls</span>):
            <span class="pl-en">initialize</span>(<span class="pl-s1">obj</span>)
        <span class="pl-k">return</span> <span class="pl-s1">obj</span>
        
<span class="pl-c"># Here's a Lazy nn.Module implementation using LazyInitMeta, calling __lazy_init__ before the first</span>
<span class="pl-c"># forward pass.</span>
    
<span class="pl-k">class</span> <span class="pl-v">LazyModule</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>, <span class="pl-s1">metaclass</span><span class="pl-c1">=</span><span class="pl-v">LazyInitMeta</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
        <span class="pl-s1">nn</span>.<span class="pl-v">Module</span>.<span class="pl-en">__init__</span>(<span class="pl-s1">self</span>)
    
    <span class="pl-k">def</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">__lazy_init__</span>:
            <span class="pl-s1">self</span>.<span class="pl-en">__lazy_init__</span>(<span class="pl-s1">call_args</span><span class="pl-c1">=</span>(<span class="pl-s1">args</span>, <span class="pl-s1">kwargs</span>))
        <span class="pl-k">return</span> <span class="pl-s1">nn</span>.<span class="pl-v">Module</span>.<span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        
<span class="pl-c"># Optionally lazy Linear module, based on the current torch implementation of nn.Linear.</span>

<span class="pl-k">class</span> <span class="pl-v">Linear</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>, <span class="pl-v">LazyModule</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">in_features</span>, <span class="pl-s1">out_features</span>, <span class="pl-s1">bias</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
        <span class="pl-v">LazyModule</span>.<span class="pl-en">__init__</span>(<span class="pl-s1">self</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">in_features</span> <span class="pl-c1">=</span> <span class="pl-s1">in_features</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">out_features</span> <span class="pl-c1">=</span> <span class="pl-s1">out_features</span>
        <span class="pl-k">if</span> <span class="pl-s1">bias</span>:
            <span class="pl-s1">self</span>.<span class="pl-s1">bias</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Parameter</span>(<span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>(<span class="pl-s1">out_features</span>))
        <span class="pl-k">else</span>:
            <span class="pl-s1">self</span>.<span class="pl-en">register_parameter</span>(<span class="pl-s">'bias'</span>, <span class="pl-c1">None</span>)
            
        <span class="pl-k">if</span> <span class="pl-s1">in_features</span> <span class="pl-c1">==</span> <span class="pl-c1">-</span><span class="pl-c1">1</span>:
            <span class="pl-s1">self</span>.<span class="pl-en">register_parameter</span>(<span class="pl-s">'weight'</span>, <span class="pl-c1">None</span>)
            ([<span class="pl-s1">input</span>], <span class="pl-s1">_</span>) <span class="pl-c1">=</span> <span class="pl-k">yield</span>  <span class="pl-c"># lazy init remainder</span>
            <span class="pl-s1">in_features</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">in_features</span> <span class="pl-c1">=</span> <span class="pl-s1">input</span>.<span class="pl-en">size</span>()[<span class="pl-c1">-</span><span class="pl-c1">1</span>]
            
        <span class="pl-s1">self</span>.<span class="pl-s1">weight</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Parameter</span>(<span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>(<span class="pl-s1">out_features</span>, <span class="pl-s1">in_features</span>))
        <span class="pl-s1">self</span>.<span class="pl-en">reset_parameters</span>()</pre></div>
<h2 dir="auto">Drawbacks</h2>
<p dir="auto">Lazy modules don’t work perfectly with everything in torch. In particular, if they’re not yet finalized (ie. -1 was passed to a dimension to be inferred, but forward hasn’t been called yet), the following behaviors might be unintuitive:</p>
<ul dir="auto">
<li>torch.save/torch.jit.script won’t work properly
<ul dir="auto">
<li>both confirmed to work fine with the sample implementation once initialization finishes</li>
</ul>
</li>
<li>module.apply()</li>
<li>any parameter initialization functions</li>
<li>module.parameters() won’t return all of the model parameters, so
<ul dir="auto">
<li>creating an optimizer/scheduler normally, ie torch.optim.Adam(module.parameters()) will potentially have unintended side-effects, as not all of the module’s eventual parameters will optimized</li>
<li>calling module.to() will move the parameters that exist, but once forward is called and new parameters are created, they will be on the default device
<ul dir="auto">
<li>this can potentially be solved with some tighter init behavior and unit tests, as there’s a finite number of components that need to be made lazy, and they can usually look at other parameters on the module and assume that users want them to be on the same device</li>
<li>alternatively it may make sense to have a concept of module-level device residency</li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 dir="auto">Alternatives</h2>
<h3 dir="auto">None instead of -1</h3>
<p dir="auto">Use the above proposal, but pass None to indicate an inferred dimension rather than -1. -1 was chosen for the proposal because it mirrors the semantics of reshape and a few other operators, but None may be more intuitive.</p>
<h3 dir="auto">Support -1 (or None) for Tensor dimensions</h3>
<p dir="auto">Instead of implementing this at the Module level with lazy modules, implement this at the Tensor level. Have Tensors be able to be lazy- or partially-initialized, and then be able to be finalized with their final dimensions. This would solve most of the above drawbacks above, as for instance we’d always be making and setting all parameters, and it would also likely simplify its integration as all known useful cases of lazy dimensions eventually boil down to passing them to Tensors. However, it’s less clear what this implementation would look like, for instance how the Tensors would generically be able to infer the parameters.</p>
<h3 dir="auto">Lazy Parameters</h3>
<p dir="auto">If this were implemented at the Parameter level, it would require pretty radically increasing the complexity of how Parameters work, but could solve most of the drawbacks of this proposal while also leaving Tensor untouched. We can flesh out this design more if there is significant feedback in that direction.</p>
<h3 dir="auto">Support via metaclass vs non-metaclass</h3>
<p dir="auto">The current proposal implementation uses a metaclass for Lazy modules to introduce a novel initialization pattern for lazy components (this same pattern could eg. also be used to solve an analogous problems for Optimizer/Scheduler in the future). However, using metaclasses always comes at a cost in Python, especially for libraries that are being built on-top-of, because multiple inheritance in the case of multiple metaclasses is a problem with no generic solution. In other words, if any libraries built on top of pytorch want to use metaclasses in their own Module subclasses, they’ll need to do non-trivial work to make sure those metaclasses are compatible with any metaclass used in pytorch.<br>
The proposal could likely be modified to, for example, use a decorator on the <strong>init</strong> function instead of a metaclass.</p>