title: Loop to get model fields, create new model instances based on API returned values
body: <p>I'm using a couple of different API calls to populate (create_or_update) a particular Model's fields. Instead of having to create a new series of create_or_update fields for every API call, I am trying to make a loop to get rid of boilerplate code. There is a small caveat in the fact that some of the fields returned by the api overlap with python reserved keywords (eg id, type).  I need to be able to pick those out and rename them in the dict during the looping comparison.</p>
  
  <ol>
  <li>Looks up the Model and gets its fields, creating a list of those fields eg:</li>
  </ol>
  
  <pre><code>['id', 'organizationId', 'name', ... ]
  </code></pre>
  
  <ol start="2">
  <li>Cycles through the API response and creates a dictionary of key:values for each response object.  If the list value is found in the dictionary, set the key: value to what is found in the API response, otherwise, set the value for the field to None. </li>
  </ol>
  
  <pre><code>{ 'id': 'N_543265', 'organizationId': '43675', *name field missing* , ...}
  </code></pre>
  
  <p>Since both 'id' and 'organizationId' are found, I want their values to be updated into the Model instance, but as 'name' is not found, I want that to be skipped</p>
  
  <p>The main issue I am having is that when this runs (it is a celery task), it tells me the following:</p>
  
  <blockquote>
    <p>database.models.Network.DoesNotExist: Network matching query does not exist.</p>
  </blockquote>
  
  <p><strong>models.py</strong></p>
  
  <pre class="lang-py prettyprint-override"><code>class Device(models.Model):
      networkId = models.ForeignKey(Network, on_delete=models.CASCADE, blank=True, null=True, related_name='network_device')
  
      name = models.CharField(max_length=100, default='', blank=True, null=True)
  organizationId = models.CharField(max_length=100, default='', blank=True)
  </code></pre>
  
  <p><strong>devices_status.py</strong></p>
  
  <pre><code>...**API call happens here**
  
      result = response.text
      job = json.loads(result)
  
      l = []
      s = []
      x = Device._meta.get_fields()
      for f in x:
          fields = []
          try:
              l.append(str(f).split('.'))
  
              for a in l:
                  s.append(a[-1])
  
              for q in s:
                  if '&lt;' in q:
                      continue
                  elif '&gt;' in q:
                      continue
                  else:
                      fields.append(q)
          except AttributeError as e:
              print(e)
              pass
  
      for line in job:
          d = {}
          for k, v in line.items():
              if k in fields:
                  d[k] = v
              else:
                  continue
  
          # Have to delete this from the auto field pull otherwise it gets 
          # confused with python factory vars
          try:
              del d['type']
          except KeyError as e:
              print('Delete type: {}'.format(e))
              pass
  
          try:
              Device.objects.update_or_create(
                  networkId = Network.objects.get(networkId=d['networkId']),
                  name = d['name'],
                  serial = d['organizationId'],
  
                  defaults = d
              )
          except KeyError as e:
              print(e)
              pass
  
      return result
  </code></pre>
  
  <p>Basically, I want my model instance to populate based on comparing fields returned in the API, to avoid throwing a KeyError, and without having a long list of try/except to manually fish out fields that may or may not exist in the API data.  </p>
  
  <p>Any assistance would be great, even if it is more high level about the way I've gone about the problem.  I think a lot of people could gain from this snippet if i can get it working correctly.</p>
  
  <p><strong><em>EDIT</em></strong></p>
  
  <p>I managed to find a solution to this problem, so I have detailed it here for anyone interested.  I have found this solution saves me a lot of headaches when importing data to the database from an API with varying fields.</p>
  
  <p>I will use a shortened version of data returned from the Meraki dashboard API (<a href="https://dashboard.meraki.com/api_docs" rel="nofollow noreferrer">https://dashboard.meraki.com/api_docs</a>):</p>
  
  <pre><code>...**API returned data**
  
  [
      {
          "lat": 1.0,
          "lng": 1.0,
          "address": "123 fake street",
          "serial": "2222-2222-2222",
          "mac": "22:22:22:22:22:22",
          "lanIp": "10.1.1.2",
          "tags": "tag1 tag2",
          "networkId": "5678",
          "name": "accesspoint01",
          "model": "ap01"
      },
      {
          "lat": 2.0,
          "lng": 2.0,
          "address": "123 fake street",
          "serial": "5555-5555-5555",
          "mac": "44:44:44:44:44:44",
          "wan1Ip": "1.1.1.1",
          "wan2Ip": null,
          "lanIp": "10.10.10.10",
          "networkId": "2468",
          "name": "gateway01",
          "model": "gw01"
      },
  ]
  ...
  </code></pre>
  
  <p>This is the type of API data we will expect to process and add into the database.</p>
  
  <p><strong>models.py</strong></p>
  
  <pre class="lang-py prettyprint-override"><code>class Device(models.Model):
      # FK
      parent_network = models.ForeignKey(Network, on_delete=models.CASCADE, blank=True, null=True, related_name='network_device')
  
      name = models.CharField(max_length=100, default='', blank=True, null=True)
      lat = models.CharField(max_length=100, default='', blank=True, null=True)
      lng = models.CharField(max_length=100, default='', blank=True, null=True)
      tags = models.CharField(max_length=200, default='', blank=True, null=True)
  
      serial = models.CharField(max_length=100, default='', blank=True, null=True)
      mac = models.CharField(max_length=100, default='', blank=True, null=True)
      publicIp = models.CharField(max_length=100, default='', blank=True, null=True)
      lanIp = models.CharField(max_length=100, default='', blank=True, null=True)
  
      wan1Ip = models.CharField(max_length=25, default='', blank=True, null=True)
      wan2Ip = models.CharField(max_length=25, default='', blank=True, null=True)
  
      model = models.CharField(max_length=100, default='', blank=True, null=True)
      address = models.CharField(max_length=100, default='', blank=True, null=True)
  
      created_at = models.DateTimeField(auto_now_add=True, verbose_name='Created')
      updated_at = models.DateTimeField(auto_now=True, verbose_name='Last Updated')
  
      def __str__(self):
          return str(self.name)
  
      class Meta:
          ordering = ['name']
  </code></pre>
  
  <p><strong>device.py</strong> (I have several similar functions in a folder, named after their function for ease of management)</p>
  
  <pre class="lang-py prettyprint-override"><code>def device(apikey):
  
      excludes = ['serial', 'parent_network']
  
  ## Once the request has been made, the returned data is assigned
      ...
      response = requests.request("GET", url, headers=headers)
      result = response.text
      job = json.loads(result)
  
      # Create the dictionary that will be used in defaults of update_or_create
      d = {}
      # Iterate over the key:value items returned from the API call
      for k, v in job.items():
          try:
              # This part checks the keys in the returned dataset to create dictionary entries based on those returned values
              if k in excludes:
                  continue
              else:
                  d[k] = v
              d['parent_network'] = Network.objects.get(parent_network=job['networkId'])
          except Exception as e:
              print('[*] device.py - Dictionary error: {}'.format(e))
              continue
      # Add the object to the database, based on the serial we return from the API
      try:
          Device.objects.update_or_create(
              serial=job['serial'],
              defaults=d,
          )
      except KeyError as e:
          print('[*] device.py - Device KeyError: {}'.format(e))
          pass
  </code></pre>
  
  <p>Why have an array excluding some items from the dictionary???</p>
  
  <p><strong>'serial'</strong> - Excluded because we want to use this as our unique field to search for an object to update
  <strong>'parent_network'</strong> - This is a foreign key field, so this has to be fed a 'Network' object</p>
  
  <p>Now if we have other similar functions that have to deal with API data, we have a 
  bit of a template that we can copy.  All we need to do to apply it to a new 
  set of API data is change the 'excludes' array and process any ForeignKey fields 
  outside of the k, v loop.</p>
  
  <p>The next thing to do would be to turn this into a fully re-usable vanilla function 
  that can be passed all the required variables but unfortunately, that is out of my 
  range of skills.  It would be great to see someone else do this if possible.</p>
  
  <p>I hope this has been helpful.  <em>Thanks for reading!</em></p>