title: python list n files then next n files in a directory and Map it to a mapper function
body: <p>I have a directory in which I have around hundred thousands of text files.<br>
  Python code creates a list of names of this files,  </p>
  
  <pre><code>listoffiles = os.listdir(directory)
  </code></pre>
  
  <p>I break this <code>listoffiles</code> with <code>lol</code> function in <strong>64</strong> parts</p>
  
  <pre><code>lol = lambda lst, sz: [lst[i:i+sz] for i in range(0, len(lst), sz)]
  partitioned_listoffiles = lol(listoffiles, 64)
  </code></pre>
  
  <p>Then I pool it to <strong>2</strong> processes</p>
  
  <pre><code>pool = Pool(processes=2,)
  single_count_tuples = pool.map(Map, partitioned_listoffiles)
  </code></pre>
  
  <p>In <code>Map</code> function I read those files and do further processing</p>
  
  <p>My problem is this code works fine if I do it for small folder with thousands of files. Large directories it runs out of memory. How should I solve this issue. Can I read first n files and then next n files and create <code>listoffiles</code> and process this steps in for loop. </p>