title: Python: request to website doesn't gives the html I need in any cases
body: <p>Based on my question <a href="https://stackoverflow.com/questions/70116360/python-request-to-website-doesnt-work-in-any-cases/70116782#70116782">here</a> I have some further question with requests on website finance.yahoo.com.</p>
  <p>My request without User-Agent request gives me the html code I want to collect some data from the website.</p>
  <p>The call with 'ALB' as parameter works fine, I get the requested data:</p>
  <pre><code>import bs4 as bs
  import requests
  def yahoo_summary_stats(stock):
      response = requests.get(f&quot;https://finance.yahoo.com/quote/{stock}&quot;)
      #response = requests.get(f&quot;https://finance.yahoo.com/quote/{stock}&quot;, headers={'User-Agent': 'Custom user agent'})
  
      soup = bs.BeautifulSoup(response.text, 'lxml')
  
      table = soup.find('p', {'class': 'D(ib) Va(t)'})
  
      sector = table.findAll('span')[1].text
      industry = table.findAll('span')[3].text
      print(f&quot;{stock}: {sector}, {industry}&quot;)
      return sector, industry
  
  web.yahoo_summary_stats('ALB')
  </code></pre>
  <p>Output:</p>
  <pre><code>ALB: Basic Materials, Specialty Chemicals
  </code></pre>
  <p>The call <code>yahoo_summary_stats('AEE')</code> doesnt work this way, so I need to acitivate headers to request the site with success.</p>
  <p>But now with parameter<code>headers={'User-Agent': 'Custom user agent'}</code> the code doesn't work and he cannot find the paragraph p with class <code>'D(ib) Va(t)'</code>.</p>
  <p>How can I solve this problem?</p>