title: How to parallelize iteration over a big bytes-type file?
body: <p>What i have: bytes file up to 16 GByte with offset (e.g. 100 bytes).
  What i need: handle action "f" in code in a fastest way, e.g. i hope with multiprocessing.</p>
  
  <p>I tried to realise <a href="http://effbot.org/zone/wide-finder.htm" rel="nofollow noreferrer">http://effbot.org/zone/wide-finder.htm</a> this approach.
  A Multi-Threaded Python Solution from that article was two times slower then original code. A Multi-Processor Python Solution i couldn't realise, because my python level is not good enough. I read multiprocessing module description, but it didn't helped for me, i become some problems with code...</p>
  
  <pre><code>from time import perf_counter
  from random import getrandbits
  
  
  def create_byte_data(size):
      creation_start = perf_counter()
      my_by = bytes(getrandbits(8) for i in range(size))  # creates 50MB random byte data
      print('creation my_by time = %.1f' % (perf_counter() - creation_start))
      return my_by
  
  
  def write_to_file(file, data, b):
      writing_start = perf_counter()
      with open(file, "wb") as f:  # binary file creation
          for a in range(offset):
              f.write(b'0')
          # for n in range(b):  # for creating bigger files
          #     f.write(data)
          f.write(data)
      print('writing time = %.1f' % (perf_counter() - writing_start))
  
  
  def abs_pixel(pixel: bytes) -&gt; int:  # converting signed bytes to absolute (0 +127) values, and collection sum of them to "result"
      result = 0
      for a in pixel:
          if a &gt; 127:
              result += 256 - a
          else:
              result += a
      return result    
  
  
  def f(file, offset, time):  # this function must be accelerated
      sum_list = list()
      with open(file, "rb") as f:
          f.seek(offset)
          while True:
              chunk = f.read(time)
              if not chunk:
                  break
              sum_list.append(abs_pixel(chunk))
      return sum_list
  
  
  if __name__ == '__main__':
      filename = 'bytes.file'
      offset = 100
      x = 512
      y = 512
      time = 200
      fs = 2  # file size in GBytes  # for creating bigger files
      xyt = x * y * time
      b = fs*1024*1024*1024//xyt  # parameter for writing data file of size 'fs'
      my_data = create_byte_data(xyt)  # don't needed after created ones
      write_to_file(filename, my_data, b)  # don't needed after created ones
      start = perf_counter()
      result = f(filename, offset, time)  # this function must be accelerated
      print('function time = %.1f' % (perf_counter() - start))
      print(result[:10])
  </code></pre>
  
  <p>Task: make some math operations with chunk (that has length "time") and collect results to list. Files can be very big, so RAM must be not overloaded.
  Code above can create random bytes file (50 Mb for start, or bigger for further tests). I expect at least 4 time acceleration for running function "f" in comparison to the code above. Actually it needs about 6 seconds on my pc for 50 MByte bytes file and about 240 seconds for 2 GByte file.</p>