<p dir="auto">Hi,</p>
<p dir="auto">I have tried to understand the code behind the DataLoader, I think I got the global meaning.</p>
<p dir="auto"><strong>Problem</strong><br>
I have noticed that while increasing the number of workers, the number of memory used increases linearly. I recall that someone told that each worker has its own replica of the dataset. I guess this is happening because when lauching a Process, the parent environment is copied for the newly created Process. It might happen differently, If I am wrong please explain me. :)</p>
<p dir="auto"><strong>Context</strong><br>
My main concern here is to have only one instance of my dataset in memory. For example my reference dataset is composed with a 8 GB file, scaling with 10 workers, you can imagine how fast memory burns. This situation has no sense because the dataset is static.<br>
I may have misinterpreted something along the way, I might not be the first one to deal with this problem.</p>
<p dir="auto"><strong>Expected workaround</strong><br>
I already built a custom dataset and cannot break this 8 GB file into more pieces, but just using a reference to the object rather than copy would save my day.</p>
<p dir="auto">Thanks</p>
<p dir="auto">When I tried to use multi-gpu trainning.</p>
<ol dir="auto">
<li>If I wrap the model forward function as follows:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def __init__(...):
        self.operation_function = self._gaussian

def forward(self, x, z):
        output = self.operation_function(x, z)

def _gaussian(self, x, z):
        Real forward codes
        ..."><pre class="notranslate"><code class="notranslate">def __init__(...):
        self.operation_function = self._gaussian

def forward(self, x, z):
        output = self.operation_function(x, z)

def _gaussian(self, x, z):
        Real forward codes
        ...
</code></pre></div>
<p dir="auto">Then</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gpu_num = torch.cuda.device_count()
print('GPU NUM: {:2d}'.format(gpu_num))
if gpu_num &gt; 1:
    model = torch.nn.DataParallel(model, list(range(gpu_num))).cuda()"><pre class="notranslate"><code class="notranslate">gpu_num = torch.cuda.device_count()
print('GPU NUM: {:2d}'.format(gpu_num))
if gpu_num &gt; 1:
    model = torch.nn.DataParallel(model, list(range(gpu_num))).cuda()
</code></pre></div>
<p dir="auto">I get one error during the multi-gpu trainning</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="File &quot;/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py&quot;, line 468, in __call__
    result = self.forward(*input, **kwargs)
  File &quot;/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py&quot;, line 123, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File &quot;/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py&quot;, line 133, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File &quot;/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py&quot;, line 77, in parallel_apply
    raise output
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)"><pre class="notranslate"><code class="notranslate">File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 468, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 123, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 133, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
    raise output
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)
</code></pre></div>
<ol start="2" dir="auto">
<li>If I directly put all codes of _gaussian function under forward function. The multi-gpu trainning works</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def forward(self, x, z):
        Real forward codes
        ..."><pre class="notranslate"><code class="notranslate">def forward(self, x, z):
        Real forward codes
        ...
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/soumith/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/soumith">@soumith</a> Do you have any comments about this?</p>