title: Parallel execution with StackExchange.Redis?
body: <p>I have a 1M items store in <code>List&lt;Person&gt;</code> Which I'm serializing in order to insert to Redis. (2.8)</p>
  
  <p>I divide work among <code>10</code> <code>Tasks&lt;&gt;</code> where each takes its own section ( <em><code>List&lt;&gt;</code> is thread safe for readonly</em> ( <em><a href="http://msdn.microsoft.com/en-us/library/6sh2ey19%28v=vs.110%29.aspx" rel="nofollow noreferrer">It is safe to perform multiple read operations on a List</a></em>)</p>
  
  <p><strong>Simplification :</strong> </p>
  
  <p>example:</p>
  
  <p>For  <code>ITEMS=100</code>, <code>THREADS=10</code> , each <code>Task</code> will capture its own PAGE and deal with the relevant range.</p>
  
  <p>For exaple : </p>
  
  <pre><code>void Main()
  {
      var ITEMS=100;
      var THREADS=10;
      var PAGE=4;
  
      List&lt;int&gt; lst = Enumerable.Range(0,ITEMS).ToList();
  
      for (int i=0;i&lt; ITEMS/THREADS ;i++)
      {
        lst[PAGE*(ITEMS/THREADS)+i].Dump();
      }
  }
  </code></pre>
  
  <ul>
  <li><code>PAGE=0</code> will deal with : <code>0,1,2,3,4,5,6,7,8,9</code></li>
  <li><code>PAGE=4</code> will deal with : <code>40,41,42,43,44,45,46,47,48,49</code></li>
  </ul>
  
  <p>All ok.</p>
  
  <p>Now back to SE.redis.</p>
  
  <p>I wanted to implement this pattern and so <a href="https://i.stack.imgur.com/viRIl.jpg" rel="nofollow noreferrer">I did</a> : (with <code>ITEMS=1,000,000</code>)</p>
  
  <p><img src="https://i.stack.imgur.com/viRIl.jpg" alt="enter image description here"></p>
  
  <p>My testing : </p>
  
  <p>(Here is <code>dbsize</code> checking  each second) : </p>
  
  <p><img src="https://i.stack.imgur.com/9TCb7.jpg" alt="enter image description here"></p>
  
  <p>As you can see , 1M records were added via 10 threads.</p>
  
  <p>Now , I don't know if it's fast but , when I change ITEMS from <code>1M</code> to <code>10M</code> --  things get <em>really</em> slow and I get exception : </p>
  
  <p>The exception is on the <code>for</code> loop.</p>
  
  <blockquote>
    <p>Unhandled Exception: System.AggregateException: One or more errors
    occurred. ---</p>
    
    <blockquote>
      <p>System.TimeoutException: Timeout performing SET urn:user>288257, inst: 1, queu e: 11, qu=0, qs=11, qc=0, wr=0/0, in=0/0    at
      StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message
      messa ge, ResultProcessor<code>1 processor, ServerEndPoint server) in
      c:\TeamCity\buildAgen
      t\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\ConnectionMultip
      lexer.cs:line 1722    at
      StackExchange.Redis.RedisBase.ExecuteSync[T](Message message,
      ResultProces sor</code>1 processor, ServerEndPoint server) in
      c:\TeamCity\buildAgent\work\58bc9a6df
      18a3782\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 79
      ... ... Press any key to continue . . .</p>
    </blockquote>
  </blockquote>
  
  <p><strong>Question:</strong></p>
  
  <ul>
  <li>Is my way of  dividing work  is the RIGHT way  (fastest)</li>
  <li>How can I get things faster ( a sample code would be much appreciated)</li>
  <li>How can I resolve this exception?</li>
  </ul>
  
  <p>Related info : </p>
  
  <p><code>&lt;gcAllowVeryLargeObjects enabled="true" /&gt;</code> Is present in App.config ( otherwise i'm getting outOfmemoryException ) , also - build for x64bit, I have 16GB ,  , ssd drive , i7 cpu).</p>