title: python beautiful soup web crawling with json
body: <p>I'm new to beautifulsoup in python and I"m trying to extract certain information from a website. In detail, the url and the title.</p>
  
  <p>I use beautifulsoup to extract the json which I successfully did but I´m unsure about the next steps, how to get the url and title</p>
  
  <p>I did not manage to extract the desired information yet. I hope you guys can help me out</p>
  
  <p>That is my logic so far:</p>
  
  <pre><code>import json
  import requests
  from bs4 import BeautifulSoup
  from urllib.request import urlopen
  import urllib.request
  
  session = requests.Session()
  session.cookies.get_dict()
  url = 'http://www.citydis.com/'
  headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1)     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
  response = session.get(url, headers=headers)
  
  soup = BeautifulSoup(response.content, "html.parser")
  metaConfig = soup.find("meta",  property="configuration")
  metaConfigTxt = metaConfig["content"]
  csrf = json.loads(metaConfigTxt)["pageToken"]
  
  
  jsonUrl = "https://www.citydis.com/s/results.json?&amp;q=London&amp;     customerSearch=1&amp;page=0"
  headers.update({'X-Csrf-Token': csrf})
  response = session.get(jsonUrl, headers=headers)
  print(response.content)
  </code></pre>
  
  <p>And that is the output:</p>
  
  <pre><code>b'{"searchResults":{"customer":null,"signupUrl":"\\/signup\\/?pos=activityCard","isMobile":false,"tours":[{"tourId":5459,"title":"Ticket f\\u00fcr Coca-Cola London Eye 4D-Erlebnis","url":"https:\\/\\/www.getyourguide.de\\/london-l57\\/ohne-anstehen-edf-london-eye-4d-erlebnis-t5459\\/","price":{"original":"27,10\\u00a0\\u20ac","min":"27,10\\u00a0\\u20ac","type":"individual"},"horizontalImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-70.jpg","horizontalAlternativeImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-85.jpg","verticalImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-92.jpg","mobileImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-53.jpg","horizontalSlimImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-67.jpg","highlightedDetailedImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-91.jpg","smallDescription":"Sehen Sie London aus einer anderen Perspektive vom London Eye aus und genie\\u00dfen Sie beim neuen 4D-Erlebnis einen bahnbrechenden 3D-Film mit\\u2026","description":"Sehen Sie London aus einer anderen Perspektive vom London Eye aus und genie\\u00dfen Sie beim neuen 4D-Erlebnis einen bahnbrechenden 3D-Film mit spektakul\\u00e4ren Spezialeffekten, einschlie\\u00dflich Wind und Nebel. Genie\\u00dfen Sie au\\u00dferdem bevorzugten Einlass am Eingang.","isBestseller":false,"isFeatured":false,"languageIds":[],"hasDeal":false,"dealMaxPercentage":0,"isBoostedNewTour":false,"hasBanner":false,"hasRibbon":false,"priceTag":true,"detailsLink":false,"isCertifiedPartner":true,"hasFencedDiscountDeal":false,"hasFreeCancellation":false,"hasRating":true,"averageRating":"4,5","totalRating":1633,"totalRatingTitle":"1633 Bewertungen","averageRatingClass":"45","ratingLink":"","ratingStyleModifier":"","ratingStarsClasses":"","ratingTitle":"Bewertung: 4,5 von 5","hasDuration":true,"duration":"40 Minuten","displayAbstract":true,"displayDuration":true,"displayDate":false,"displayWishlist":false,"displayRemoveButton":false,"hasDiscountedRecommendation":false,"hideImage":false,"isSkipTheLine":false,"likelyToSellOutBadge":true,"isPromoted":false,"isSpecialOffer":false,"experiments":{"hasRatingsExperiment":false,"numericRatingLabel":"Basierend auf 1633 Bewertungen","verticalImageForPriceSegmentation":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-412120-150.jpg"},"id":"searchResults","activityCardVersion":"horizontal","limit":false,"likelyToSellOutExperiment":{"deviceDetector":{}},"hasNumericReviews":true,"resultSetPosition":0,"activityCardStyle":"plain","highlightedOrientation":"horizontal"},{"tourId":51268,"title":"Bustransfer: Flughafen Stansted - Stadtzentrum London","url":"https:\\/\\/www.getyourguide.de\\/london-l57\\/bustransfer-flughafen-stansted-stadtzentrum-london-t51268\\/","price":{"original":"9,43\\u00a0\\u20ac","min":"9,43\\u00a0\\u20ac","type":"individual"},"horizontalImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-70.jpg","horizontalAlternativeImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-85.jpg","verticalImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-92.jpg","mobileImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-53.jpg","horizontalSlimImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-67.jpg","highlightedDetailedImageUrl":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-91.jpg","smallDescription":"Beginnen oder beenden Sie Ihren Aufenthalt in London mit dem praktischen Bustransfer zwischen dem Flughafen Stansted und dem Stadtzentrum London.\\u2026","description":"Beginnen oder beenden Sie Ihren Aufenthalt in London mit dem praktischen Bustransfer zwischen dem Flughafen Stansted und dem Stadtzentrum London. Sparen Sie sich die Fahrt mit \\u00f6ffentlichen Verkehrsmitteln und erreichen Sie London schnell und bequem.","isBestseller":false,"isFeatured":false,"languageIds":[],"hasDeal":false,"dealMaxPercentage":0,"isBoostedNewTour":false,"hasBanner":false,"hasRibbon":false,"priceTag":true,"detailsLink":false,"isCertifiedPartner":false,"hasFencedDiscountDeal":false,"hasFreeCancellation":true,"hasRating":true,"averageRating":"4,4","totalRating":541,"totalRatingTitle":"541 Bewertungen","averageRatingClass":"45","ratingLink":"","ratingStyleModifier":"","ratingStarsClasses":"","ratingTitle":"Bewertung: 4,4 von 5","hasDuration":true,"duration":"60 Minuten \\u2013 90 Minuten","displayAbstract":true,"displayDuration":true,"displayDate":false,"displayWishlist":false,"displayRemoveButton":false,"hasDiscountedRecommendation":false,"hideImage":false,"isSkipTheLine":false,"likelyToSellOutBadge":true,"isPromoted":false,"isSpecialOffer":false,"experiments":{"hasRatingsExperiment":false,"numericRatingLabel":"Basierend auf 541 Bewertungen","verticalImageForPriceSegmentation":"https:\\/\\/cdn.getyourguide.com\\/img\\/tour_img-451822-150.jpg"}
  </code></pre>
  
  <p>What I would like to get out is the title and url only. For example:</p>
  
  <pre><code>title":"Ticket f\\u00fcr Coca-Cola London Eye 4D-Erlebnis","url":"https:\\/\\/www.getyourguide.de\\/london-l57\\/ohne-anstehen-edf-london-eye-4d-erlebnis-t5459
  </code></pre>
  
  <p>Any feedback much appreciated</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Thanks to the feedback I was able to solve the problem. </p>
  
  <p>I´m now able to get the desired result but now I have the issue that I´m just getting one result back instead of all available:</p>
  
  <pre><code>js_dict = (json.loads(response.content.decode('utf-8')))
  
  url = (js_dict['searchResults']["tours"][0]["url"])
  print(url)
  
  title = (js_dict['searchResults']["tours"][0]["title"])
  print(title)
  
  price = (js_dict['searchResults']["tours"][0]["price"]["original"])
  print(price)
  </code></pre>
  
  <p>Output is the following one:</p>
  
  <pre><code>https://www.citydis.de/london-l57/ohne-anstehen-edf-london-eye-4d-erlebnis-t5459/
  Ticket für Coca-Cola London Eye 4D-Erlebnis
  27,10 €
  </code></pre>
  
  <p>I would like to get all the titles, prices and urls back of the sightseeings which are in the JSON. I tried with the for loop but somehow it does not work.</p>
  
  <p>Any feedback appreciated</p>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p>Found a solution:</p>
  
  <pre><code>jsonUrl = "https://www.citydis.com/s/results.json?&amp;q=London&amp; customerSearch=1&amp;page=0"
  headers.update({'X-Csrf-Token': csrf})
  response = session.get(jsonUrl, headers=headers)
  js_dict = (json.loads(response.content.decode('utf-8')))
  
  for item in js_dict:
      headers = js_dict['searchResults']["tours"]
      prices = js_dict['searchResults']["tours"]
      urls = js_dict['searchResults']["tours"]
  
  
  for title, price, url in zip(headers, prices, urls):
  
      title_final = title.get("title")
      url_final = url.get("url")
      price_final = price.get("price")["original"]
      print("Header: " + title_final + " | " + "Deeplink: " + url_final + " | " + "Price: " + price_final)
  </code></pre>