title: POST requests with JSON data, how to break it up into chunks
body: <p>I have this script to send many (like 100,000++) records from a table. So I would like to have some error handling, and break up the data payload into smaller chunks and send via POST request. My current script works sometimes, but most times return that heap space error.</p>
  <pre><code>import json
  import requests
  from multiprocessing import Pool
  
  url = &quot;some url&quot;
  header = {&quot;Content-Type&quot;: &quot;application/json&quot;, 'Accept':'application/json'}
  api_key=&quot;some key&quot;
  
  def send_request(data):
      try:
        res = requests.post(url, headers=header, json={&quot;api_key&quot;: api_key, &quot;attributes&quot;: data})
        print(res.status_code)
        print(res.json())
      except requests.exceptions.RequestException as e:
        #catastrophic error. bail.
        print(e)
  
  def chunks(data):
      for i in range(0, len(data), 50):
          yield data[i:i + 50]
  
  
  
  p = Pool(8)
  p.map(send_request, chunks(json.loads(data)))
  </code></pre>