title: Get gzip file from endpoint, extract data and add to database improvements
body: <p>I currently have a script that fires a request to an API endpoint which returns a <code>csv.gzip</code> file - which roughly contains 75,000 rows and 15 columns. I download this files to the webserver disk storage, unzip the file to <code>.csv</code> and then loop through every row and add the data into my database. Finally deleting the files saved to disk. The process currently takes between 5 and 10 minutes to complete.</p>
  <p>I'm sure there is areas of improvement but not sure how to implement them. Some of them is:</p>
  <ol>
  <li>Save csv data to variable rather than on disk.</li>
  <li>Bulk import the data into my database.</li>
  </ol>
  <p>I'm sure there are other improvements to be made, so any advice would be appreciated.</p>
  <pre class="lang-py prettyprint-override"><code>response = oauth.get(realm)
  
  content = ET.fromstring(response.content)
  coded_string = (content.find('.//pricefile'))
  decoded_string = base64.b64decode(coded_string.text)
  with open('test.csv.gzip', 'wb') as f:
      f.write(decoded_string)
      with gzip.open('test.csv.gzip', 'rb') as f_in:
          with open('test.csv', 'wb') as f_out:
              shutil.copyfileobj(f_in, f_out)
  
  with open('test.csv') as f:
      reader = csv.reader(f)
      next(reader)
      for row in reader:
          pricing.objects.update_or_create(
              product_id=row[0],
              date=datetime.now(),
  
              defaults={
                  'average_price': Decimal(row[1] or 0),
                  'low_price': Decimal(row[2] or 0),
                  'high_price': Decimal(row[3] or 0),
                  ...
              })
  
  os.remove('test.csv')
  os.remove('test.csv.gzip')
  </code></pre>