title: Web-scraping with Python to extract microdata for each page from a sitemap.xml
body: <p>I'm trying to extract name, brand, prices, stock microdata from pages extracted from sitemap.xml
  But I'm blocked with the following step, thank you for helping me as I'm a newbie I can't understand the blocking element</p>
  <ul>
  <li>Scrape the sitemap.xml to have list of urls : OK</li>
  <li>Extract the metadata : OK</li>
  <li>Extract the product schema : OK</li>
  <li>Extract the products not OK</li>
  <li>Crawl the site and store the products not OK</li>
  </ul>
  <hr />
  <ul>
  <li>Scrape the sitemap.xml to have list of urls : OK</li>
  </ul>
  <pre><code>import pandas as pd
  import requests
  import extruct
  from w3lib.html import get_base_url
  import urllib.request
  from urllib.parse import urlparse
  from bs4 import BeautifulSoup
  import advertools as adv
  proximus_sitemap = adv.sitemap_to_df('https://www.proximus.be/iportal/sitemap.xml')
  proximus_sitemap = proximus_sitemap[proximus_sitemap['loc'].str.contains('boutique')]
  proximus_sitemap = proximus_sitemap[proximus_sitemap['loc'].str.contains('/fr/')]
  </code></pre>
  <ul>
  <li>Extract the metadata : OK</li>
  </ul>
  <pre><code>def extract_metadata(url):
   
      r = requests.get(url)
      base_url = get_base_url(r.text, r.url)
      metadata = extruct.extract(r.text, 
                                 base_url=base_url,
                                 uniform=True,
                                 syntaxes=['json-ld',
                                           'microdata',
                                           'opengraph'])
      return metadata
  metadata = extract_metadata('https://www.proximus.be/fr/id_cr_apple-iphone-13-128gb-blue/particuliers/equipement/boutique/apple-iphone-13-128gb-blue.html')
  metadata
  </code></pre>
  <ul>
  <li>Extract the product schema : OK</li>
  </ul>
  <pre><code>def get_dictionary_by_key_value(dictionary, target_key, target_value):
      for key in dictionary:
          if len(dictionary[key]) &gt; 0:
              for item in dictionary[key]:
                  if item[target_key] == target_value:
                      return item
  Product = get_dictionary_by_key_value(metadata, &quot;@type&quot;, &quot;Product&quot;)
  Product
  </code></pre>
  <ul>
  <li>Extract the products not OK =&gt; errormessage = errorkey 'offers'</li>
  </ul>
  <pre><code>def get_products(metadata):
   
      
      Product = get_dictionary_by_key_value(metadata, &quot;@type&quot;, &quot;Product&quot;)
  
      if Product:
      
          products = []
  
          for offer in Product['offers']['offers']:
              product = {
                  'product_name': Product.get('name', ''),
                  'brand': offer.get('description', ''),
                  'availability': offer.get('availability', ''),
                  'lowprice': offer.get('lowPrice', ''),
                  'highprice': offer.get('highPrice', ''),
                  'price': offer.get('price', ''),
                  'priceCurrency': offer.get('priceCurrency', ''),
              }
  
              products.append(product)
  
          return products
  </code></pre>
  <ul>
  <li>Crawl the site and store the products not OK as blocked during previous step</li>
  </ul>
  <pre><code>def scrape_products(proximus_sitemap, url='url'):
   
  
      df_products = pd.DataFrame(columns = ['product_name', 'brand', 'name', 'availability', 
                                            'lowprice', 'highprice','price','priceCurrency'])
  
      for index, row in proximus_sitemap.iterrows(): 
          
          metadata = extract_metadata(row[url])          
          products = get_products(metadata)
          
          if products is not None:
              for product in products:             
                  df_products = df_products.append(product, ignore_index=True)
  
      return df_products
  df_products = scrape_products(proximus_sitemap, url='loc')
  df_products.to_csv('patch.csv', index=False)
  df_products.head()
  </code></pre>