title: Python Parallel-process dictionary translations in a single text file
body: <p>I have climate sensor data that is generated like this from a sensor hub and stored in a text file periodically:</p>
  
  <pre><code>{"Time":1541203508.45,"Tc":25.4,"Hp":33}
  {"Time":1541203508.45,"Tc":25.2,"Hp":32}
  {"Time":1541203508.45,"Tc":25.1,"Hp":31}
  {"Time":1541203508.45,"Tc":25.2,"Hp":33}
  </code></pre>
  
  <p>I'm doing a lot of list lookups (binning) in a for loop like this:</p>
  
  <pre><code>#generate sample data
  sensor_data={x:{"Time":15e6+x,"Tc":random.randint(20e3,50e3)/1000.0,"Hp":random.randint(0,100e3)/1000.0} for x in range(int(1e5))}
  
  #process data
  for i in sensor_data:
      #we'd normally do a json.loads(data_line) if reading from the file
      sensor_data[i]['H']=['V_Dry','Dry','Normal','Humid','V_Humid','ERR'][int(sensor_data[i]['Hp']//20)]
      sensor_data[i]['T']=['V_Cold','Cold','Normal','Hot','V_Hot','ERR'][int(sensor_data[i]['Tc']//10)]
      #.... And so on for other sensors etc
  </code></pre>
  
  <p>input:</p>
  
  <pre><code>{0: {'Hp': 20.514, 'Tc': 43.92, 'Time': 15000000.0},
   1: {'Hp': 59.332, 'Tc': 35.592, 'Time': 15000001.0},
   2: {'Hp': 19.49, 'Tc': 25.813, 'Time': 15000002.0},
   3: {'Hp': 78.644, 'Tc': 48.07, 'Time': 15000003.0},
   4: {'Hp': 3.967, 'Tc': 35.058, 'Time': 15000004.0}}
  </code></pre>
  
  <p>output:</p>
  
  <pre><code>{0: {'H': 'Dry', 'Hp': 20.514, 'T': 'V_Hot', 'Tc': 43.92, 'Time': 15000000.0},
   1: {'H': 'Normal', 'Hp': 59.332, 'T': 'Hot', 'Tc': 35.592, 'Time': 15000001.0},
   2: {'H': 'V_Dry', 'Hp': 19.49, 'T': 'Normal', 'Tc': 25.813, 'Time': 15000002.0},
   3: {'H': 'Humid', 'Hp': 78.644, 'T': 'V_Hot', 'Tc': 48.07, 'Time': 15000003.0},
   4: {'H': 'V_Dry', 'Hp': 3.967, 'T': 'Hot', 'Tc': 35.058, 'Time': 15000004.0}}
  </code></pre>
  
  <p>How can I speed up this translation?
  We usually have 100MB-1GB files of this data that we have to read from a file.</p>
  
  <p>I've tried a few things already:</p>
  
  <p>I won't multiprocess reading the file because that would cause too many fetches, which I believe is incorrect.</p>
  
  <p>I've tried multiprocessing chunks of the datalines using a function that split the text file into chunks, and the multiprocessing overhead seemed to be too much.</p>
  
  <p>I've tried 4 processes each consuming a queue and 4 processes each consuming different queue using a manager.list object passed to the processes.</p>
  
  <p>In each case, just running the simple 'for' loop over the data is timed faster overall. What do I do to multiprocess/thread/anything else to be faster than the serial data processing? </p>
  
  <p>The data does not have to be in any order because we have the timestamps that we can use to sort later.
  We have a 4 core i7 at our disposal. We use ujson which seems to be faster.</p>