title: Convert Dictionary To Key and Value Tensor
body: <p>I am dealing with a large scale dictionary (2M+ key,value pairs). I want to keep the information stored in memory, not having to resort to writing to disk. I tested with other formats besides dictionaries including Pytorch Tensors and it seems that would be more efficient. I have included the code from my experiments below:</p>
  <pre><code>import torch
  import sys
  import random
  source = list(random.choices(list(range(0,10)), k=100))
  dest = list(range(100))
  ls1 = torch.Tensor([source, dest])
  dic1 = {}
  for i in range(len(source)):
      src = source[i]
      dst = dest[i]
      if src not in dic1:
          dic1[src] = []
      dic1[src].append(dst)
  
  sys.getsizeof(ls1)
  # &gt;&gt;&gt; 64
  sys.getsizeof(dic1)
  # &gt;&gt;&gt; 364
  </code></pre>
  <p>Based on this experiment, if storing into a PyTorch tensor is the most memory efficient, what would be the fastest way to convert from a dictionary to a tensor? My use case dictates using a loop that calls a function that returns a large dictionary. While a single dictionary returned by the function fits in memory, combining it with another dictionary would cause a memory issue. My solution is at every iteration of the loop to take the dictionary, convert it into a PyTorch tensor, and combine it with the previous iterations' tensor.</p>