title: python - How to resolve KeyError when pulling from a json dictionary
body: <p>I'm working on a project where I'm utilizing the Foursquare API to pull venue information from locations around Portland. I've been having an issue when trying to pull information from the json dictionary and structuring it into a dataframe using a custom function I wrote. For some reason, the function I wrote is flagging a KeyError on the 'groups' section in the dictionary, even though there are items in it and it is present in the dictionary. I had a hunch that the KeyError could be triggered due to some of the groups in the dictionary not having any items in them when. </p>
  
  <p>ex: (look at the 'items' subgroup towards the bottom and you'll see there is an empty list. My function is looking for information in this subgroup, so maybe that's what is flagging the error,)</p>
  
  <pre><code>{'meta': {'code': 200, 'requestId': '5e4330219fcb92001bd8a847'},
   'response': {'warning': {'text': 'There aren\'t a lot of results for "(\'Bookstore\', \'Comic Shop\', \'Used Bookstore\', \'Library\')." Try something more general, reset your filters, or expand the search area.'},
    'headerLocation': 'University Park',
    'headerFullLocation': 'University Park, Portland',
    'headerLocationGranularity': 'neighborhood',
    'query': 'bookstore comic shop used bookstore library',
    'totalResults': 0,
    'suggestedBounds': {'ne': {'lat': 45.5830000045, 'lng': -122.72538279609336},
     'sw': {'lat': 45.573999995499996, 'lng': -122.73821720390666}},
    'groups': [{'type': 'Recommended Places',
      'name': 'recommended',
      'items': []}]}}
  </code></pre>
  
  <p>compared to this, which would be what a "normal" dictionary from the API request would look like:</p>
  
  <pre><code>{'meta': {'code': 200, 'requestId': '5e4335769da7ee001b203b56'},
   'response': {'warning': {'text': 'There aren\'t a lot of results for "(\'Bookstore\', \'Comic Shop\', \'Used Bookstore\', \'Library\')." Try something more general, reset your filters, or expand the search area.'},
    'headerLocation': 'Cathedral Park',
    'headerFullLocation': 'Cathedral Park, Portland',
    'headerLocationGranularity': 'neighborhood',
    'query': 'bookstore comic shop used bookstore library',
    'totalResults': 2,
    'suggestedBounds': {'ne': {'lat': 45.5928000045, 'lng': -122.7515716757994},
     'sw': {'lat': 45.583799995499994, 'lng': -122.76440832420062}},
    'groups': [{'type': 'Recommended Places',
      'name': 'recommended',
      'items': [{'reasons': {'count': 0,
         'items': [{'summary': 'This spot is popular',
           'type': 'general',
           'reasonName': 'globalInteractionReason'}]},
        'venue': {'id': '5632a8dc498e5636877e4fa3',
         'name': 'Comic Cave PDX',
         'location': {'address': '1920 N. Kirkpatrick',
          'crossStreet': 'Denver',
          'lat': 45.59002437301252,
          'lng': -122.75552067488051,
          'labeledLatLngs': [{'label': 'display',
            'lat': 45.59002437301252,
            'lng': -122.75552067488051}],
          'distance': 271,
          'postalCode': '97217',
          'cc': 'US',
          'city': 'Portland',
          'state': 'OR',
          'country': 'United States',
          'formattedAddress': ['1920 N. Kirkpatrick (Denver)',
           'Portland, OR 97217',
           'United States']},
         'categories': [{'id': '52f2ab2ebcbc57f1066b8b18',
           'name': 'Comic Shop',
           'pluralName': 'Comic Shops',
           'shortName': 'Comic Shop',
           'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/shops/comic_',
            'suffix': '.png'},
           'primary': True}],
         'photos': {'count': 0, 'groups': []}},
        'referralId': 'e-0-5632a8dc498e5636877e4fa3-0'},
       {'reasons': {'count': 0,
         'items': [{'summary': 'This spot is popular',
           'type': 'general',
           'reasonName': 'globalInteractionReason'}]},
        'venue': {'id': '5d617b8e228cfc0008fb1f9f',
         'name': 'Two Rivers Bookstore',
         'location': {'address': '8836 N Lombard St',
          'lat': 45.591434,
          'lng': -122.75648600000001,
          'labeledLatLngs': [{'label': 'display',
            'lat': 45.591434,
            'lng': -122.75648600000001}],
          'distance': 368,
          'postalCode': '97203',
          'cc': 'US',
          'city': 'Portland',
          'state': 'OR',
          'country': 'United States',
          'formattedAddress': ['8836 N Lombard St',
           'Portland, OR 97203',
           'United States']},
         'categories': [{'id': '4bf58dd8d48988d114951735',
           'name': 'Bookstore',
           'pluralName': 'Bookstores',
           'shortName': 'Bookstore',
           'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/shops/bookstore_',
            'suffix': '.png'},
           'primary': True}],
         'photos': {'count': 0, 'groups': []}},
        'referralId': 'e-0-5d617b8e228cfc0008fb1f9f-1'}]}]}}
  </code></pre>
  
  <p>I added an Except clause recently which allowed the function to execute, but when I used it every item was flagged as an exception (even the pulls that should have items in the dictionary). My code and the errors are below:</p>
  
  <p><strong>My code:</strong> </p>
  
  <pre><code>def getNearbyVenues(names, latitudes, longitudes, limit=500):
  
      venues_list=[]
      for name, lat, lng in zip(names, latitudes, longitudes):
          print(name)
          url = 'https://api.foursquare.com/v2/venues/explore?&amp;client_id={}&amp;client_secret={}&amp;ll={},{}&amp;v={}&amp;query={}&amp;radius={}&amp;limit={}'.format(
              CLIENT_ID, 
              CLIENT_SECRET, 
              VERSION,
              lat,
              lng,
              query, 
              radius, 
              LIMIT)
  
          try:
          results = requests.get(url, "none").json()['response']['groups'][0]['items']
  
  
              venues_list.extend([(
                  name, 
                  lat, 
                  lng, 
                  v['venue']['name'], 
                  v['venue']['location']['lat'], 
                  v['venue']['location']['lng'],  
                  v['venue']['categories'][0]['name']) for v in results])
  
          except KeyError:
              venues_list.extend([(
                 name,
                 lat,
                 lng,
                 np.nan,
                 np.nan,
                 np.nan,
                 np.nan)]) 
  
      nearby_venues = pd.DataFrame(venues_list, columns = ['Neighborhood', 
                    'Neighborhood Latitude', 
                    'Neighborhood Longitude', 
                    'Venue', 
                    'Venue Latitude', 
                    'Venue Longitude', 
                    'Venue Category'])
  
      return(nearby_venues)
  
  </code></pre>
  
  <p>(As a side note, what is the 0 in the results pull above (['responses']['groups'][0]['items]) doing exactly? Borrowed this section of code from the course that I'm working on. </p>
  
  <pre><code>---------------------------------------------------------------------------
  KeyError                                  Traceback (most recent call last)
  &lt;ipython-input-16-66be6cf8c7d3&gt; in &lt;module&gt;
        1 PDX_venues = getNearbyVenues(names=PDX_NeighDF['Neighborhood'],
        2                              latitudes=PDX_NeighDF['Latitude'],
  ----&gt; 3                              longitudes=PDX_NeighDF['Longitude']
        4                              )
        5 
  
  &lt;ipython-input-15-dab42c6d540d&gt; in getNearbyVenues(names, latitudes, longitudes, limit)
       15 
       16         #try:
  ---&gt; 17         results = requests.get(url, "none").json()['response']['groups'][0]['items']
       18 
       19 
  
  KeyError: 'groups'
  </code></pre>
  
  <p>This is the function trying to build the dataframe:</p>
  
  <pre><code>PDX_venues = getNearbyVenues(names=PDX_NeighDF['Neighborhood'],
                               latitudes=PDX_NeighDF['Latitude'],
                               longitudes=PDX_NeighDF['Longitude']
                               )
  
  PDX_venues.head()
  
  output: (structured in a pandas df normally)
  
      Neighborhood    Neighborhood Latitude   Neighborhood Longitude  Venue   Venue Latitude  Venue Longitude Venue Category
  0   CATHEDRAL PARK  45.58830    -122.75799  NaN NaN NaN NaN
  1   UNIVERSITY PARK 45.57850    -122.73180  NaN NaN NaN NaN
  2   PIEDMONT    45.56510    -122.66820  NaN NaN NaN NaN
  3   WOODLAWN    45.56970    -122.65240  NaN NaN NaN NaN
  4   ARBOR LODGE 45.57354    -122.69240  NaN NaN NaN NaN
  </code></pre>
  
  <p>I greatly appreciate any help! Let me know if you need any more info or background on the code and I'll be happy to provide more.</p>