title: Webscraping - html issue
body: <p>Ive been trying this code and getting some success but cannot figure out the next step</p>
  <pre><code>import pandas as pd
  import requests
  from termcolor import colored
  from bs4 import BeautifulSoup
  import requests
  import lxml.html as lh
  
  import pprint
  import json
  
  
  
  url2 = &quot;https://www.procyclingstats.com/rankings.php&quot;
  
  
  
  print(colored('#Step1','green'))
  response = requests.get(url2)
  soup = BeautifulSoup(response.text, 'lxml')
  table = soup.find('table', {'class':'basic'})
  headers = [heading.text for heading in table.find_all('th',{&quot;class&quot;:&quot;cu600&quot;})]
  #print(headers)
  
  #why do I only get two headers here (prev, team)?
  
  
  
  
  
  response = requests.get(url2)
  dfs = pd.read_html(response.text)[0]
  #print(list(dfs))
  
  
  #with pd.option_context('display.max_rows', None, 'display.max_columns', None):  # more options can be specified also
   #  print(dfs)
     
  print(colored('#Step1','red'))
  
  
  print(colored('#Step2','green'))
  
  url3 = &quot;https://www.procyclingstats.com/rider/tadej-pogacar&quot;
  
  response = requests.get(url3)
  soup = BeautifulSoup(response.text, 'lxml')
  table2 = soup.find({'class':'class=&quot;mt10 pps&quot;'})
  #headers = [heading.text for heading in table1.find_all('th',{&quot;class&quot;:&quot;cu600&quot;})]
  #print(headers)
  
  
  
  # Usually the line below is enough
  # But for some reason returning Forbidden
  #dfs = pd.read_html(url)[0]
  
  response = requests.get(url3)
  dfs2 = pd.read_html(response.text)[0]
  
  
  #with pd.option_context('display.max_rows', None, 'display.max_columns', None):  # more options can be specified also
      #print(dfs2)
   
  
  child_soup = soup.find('h3')
      
  for i in child_soup.children:
        print(&quot;child :  &quot;, i)
      
  print('\n'*3)  
  </code></pre>
  <p>I end up with the child as Rider (result text below)</p>
  <p>Ive been trying this code and getting some success but cannot figure out the next step</p>
  <pre><code>import pandas as pd
  import requests
  from termcolor import colored
  from bs4 import BeautifulSoup
  import requests
  import lxml.html as lh
  
  import pprint
  import json
  
  
  
  url2 = &quot;https://www.procyclingstats.com/rankings.php&quot;
  
  
  
  print(colored('#Step1','green'))
  response = requests.get(url2)
  soup = BeautifulSoup(response.text, 'lxml')
  table = soup.find('table', {'class':'basic'})
  headers = [heading.text for heading in table.find_all('th',{&quot;class&quot;:&quot;cu600&quot;})]
  #print(headers)
  
  #why do I only get two headers here (prev, team)?
  
  
  
  
  
  response = requests.get(url2)
  dfs = pd.read_html(response.text)[0]
  #print(list(dfs))
  
  
  #with pd.option_context('display.max_rows', None, 'display.max_columns', None):  # more options can be specified also
   #  print(dfs)
     
  print(colored('#Step1','red'))
  
  
  print(colored('#Step2','green'))
  
  url3 = &quot;https://www.procyclingstats.com/rider/tadej-pogacar&quot;
  
  response = requests.get(url3)
  soup = BeautifulSoup(response.text, 'lxml')
  table2 = soup.find({'class':'class=&quot;mt10 pps&quot;'})
  #headers = [heading.text for heading in table1.find_all('th',{&quot;class&quot;:&quot;cu600&quot;})]
  #print(headers)
  
  
  
  # Usually the line below is enough
  # But for some reason returning Forbidden
  #dfs = pd.read_html(url)[0]
  
  response = requests.get(url3)
  dfs2 = pd.read_html(response.text)[0]
  
  
  #with pd.option_context('display.max_rows', None, 'display.max_columns', None):  # more options can be specified also
      #print(dfs2)
   
  
  child_soup = soup.find('h3')
      
  for i in child_soup.children:
        print(&quot;child :  &quot;, i)
      
  print('\n'*3)  
  </code></pre>
  <p>I end up with the child as Rider (result text below)</p>
  <pre><code>#Step2
  child :   Rider
  </code></pre>
  <p>What Im trying to capture is the 'points per speciality' and the values.</p>
  <p>There is a second question as to why I only get two tags when they all appear to have the same name?</p>
  <p><a href="https://i.stack.imgur.com/eDYcv.png" rel="nofollow noreferrer">Photo and arrow showing desired result</a></p>