<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Hi,</p>
<p dir="auto">I just got a very strange problem:</p>
<p dir="auto">I used the commands: "print(torch.<strong>version</strong>)<br>
print(torch.version.cuda)<br>
print(torch.backends.cudnn.version())", the output were:</p>
<p dir="auto">0.4.1.post2 (PyTorch version)<br>
9.0.176 (CUDA version)<br>
7.1.2 (cuDNN version)</p>
<p dir="auto">I also used the "nvcc --version", the results were:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/37693236/50538724-91617400-0b6b-11e9-8025-de8df5d3b498.PNG"><img src="https://user-images.githubusercontent.com/37693236/50538724-91617400-0b6b-11e9-8025-de8df5d3b498.PNG" alt="capture4" style="max-width: 100%;"></a></p>
<p dir="auto">It was very strange, it seems that the CUDA version installed in Linux system is 9.0.176, and the CUDA that the PyTorch needed is also 9.0.176, but, the "cuda.is_available()" still returns "<strong>False</strong>".</p>
<p dir="auto">Could anyone help me with this? Many thanks!</p>

<h2 dir="auto">Environment</h2>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 0.4.1.post2</li>
<li>OS (e.g., Linux): Linux</li>
<li>Python version: 3.6.5 : : Anaconda, Inc.</li>
<li>CUDA/cuDNN version:  9.0.176 (CUDA version); 7.1.2 (cuDNN version)</li>
</ul>
<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>
<p dir="auto">Add metadata to the state_dict to support round trip saving and loading. Specifically, if the same object appears more than once in the object hierarchy, then that fact should be captured in the state_dict representation so that an object produced via <code class="notranslate">load_state_dict</code> is equivalent to the original object.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">In a hierarchy of nn.Modules wherein one object can appear in multiple places, the state_dict shows the state in all places, but when the object is mutated via one of its attribute aliases, the state will update in all places. This is good and correct, but when you save this state_dict this information is lost; If you load the state_dict from a file, what was once a single object is now multiple objects that are completely separate in memory. This means that saving and loading in PyTorch is not round-trip equivalent.</p>
<p dir="auto">Ideally, saving an object x, and then loading x's state into a new object y should produce two objects that are completely equivalent, in other words x should equal y (besides the memory address)</p>
<p dir="auto">Here is an example in code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class A(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.x = nn.Linear(2, 2)
class B(nn.Module):
    def __init__(self, *args, a=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.a = A() if a is None else a
class C(nn.Module):
    def __init__(self, *args, a=None, b=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.a = A() if a is None else a
        self.b = B() if b is None else b"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">A</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>)
<span class="pl-k">class</span> <span class="pl-v">B</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-v">A</span>() <span class="pl-k">if</span> <span class="pl-s1">a</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span> <span class="pl-k">else</span> <span class="pl-s1">a</span>
<span class="pl-k">class</span> <span class="pl-v">C</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-v">A</span>() <span class="pl-k">if</span> <span class="pl-s1">a</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span> <span class="pl-k">else</span> <span class="pl-s1">a</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-v">B</span>() <span class="pl-k">if</span> <span class="pl-s1">b</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span> <span class="pl-k">else</span> <span class="pl-s1">b</span></pre></div>
<p dir="auto">And the actual behavior is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; a = A()
&gt;&gt;&gt; b = B(a=a)
&gt;&gt;&gt; c = C(a=a, b=b)
&gt;&gt;&gt; c.b.a is c.a
True
&gt;&gt;&gt; c.b.a.x[0][0] = 0.1111
&gt;&gt;&gt; c.a.x[0][0]
0.1111
&gt;&gt;&gt; s = c.state_dict()
&gt;&gt;&gt; c2 = C()
&gt;&gt;&gt; c2.load_state_dict(s)
&gt;&gt;&gt; c2.b.a is c2.a
False"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; a = A()
&gt;&gt;&gt; b = B(a=a)
&gt;&gt;&gt; c = C(a=a, b=b)
&gt;&gt;&gt; c.b.a is c.a
True
&gt;&gt;&gt; c.b.a.x[0][0] = 0.1111
&gt;&gt;&gt; c.a.x[0][0]
0.1111
&gt;&gt;&gt; s = c.state_dict()
&gt;&gt;&gt; c2 = C()
&gt;&gt;&gt; c2.load_state_dict(s)
&gt;&gt;&gt; c2.b.a is c2.a
False
</code></pre></div>
<p dir="auto">Ideally, the final check would produce:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; c2.b.a is c2.a
True
&gt;&gt;&gt; c2.b.a.x[0][0] = 0.2222
&gt;&gt;&gt; c2.b.a.x[0][0] == c2.a.x[0][0]
True"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; c2.b.a is c2.a
True
&gt;&gt;&gt; c2.b.a.x[0][0] = 0.2222
&gt;&gt;&gt; c2.b.a.x[0][0] == c2.a.x[0][0]
True
</code></pre></div>
<h2 dir="auto">Pitch</h2>
<p dir="auto">Using the <code class="notranslate">state_dict._metadata</code>, we could add information about children modules being equivalent. This information can be duplicated across all common ancestors. Then, either before or after loading state in <code class="notranslate">load_state_dict</code> or its helper methods, you could check<br>
<code class="notranslate">if self.b.a is not self.a: self.b.a = self.a</code></p>
<h2 dir="auto">Alternatives</h2>
<p dir="auto">[TODO?]</p>
<h2 dir="auto">Additional context</h2>