title: How to get full job descriptions from Indeed using Python and BeautifulSoup
body: <p>I need to scrape job postings from Indeed. I managed to scrape the titles and links for each job post, and now am struggling to scrape the full job descriptions of each job posting (I don't want the summary - I want each job post's full job description).</p>
  <p>My code looks like this:</p>
  <pre><code>from bs4 import BeautifulSoup
  import requests
  import pandas as pd
  import time
  
  url = 'https://www.indeed.com/jobs?q=data+engineer&amp;l=United+States'
  
  response = requests.get(url) 
  data = response.text
  soup = BeautifulSoup(data, 'html.parser')
  jobs = soup.find_all('div',{'class':'jobsearch-SerpJobCard'})
    
  for job in jobs:
      title = job.find('a',{'class':'jobtitle'}).text
      link1 = job.find('a',{'class':'jobtitle'}).get('href')
      link = 'https://www.indeed.com' + link1
      
      #for each JOB's webpage, you need to connect to the link first:
      job_response = requests.get(link)
      job_data = response.text
      job_soup = BeautifulSoup(job_data, 'html.parser')
      
      job_description_tag = job_soup.find('div',{'id':'jobDescriptionText'})
      job_description = job_description_tag.text if job_description_tag else &quot;N/A&quot;
      
      print('Job Title:', title, '\nLink:', link, '\nJob Description:', job_description, '\n---')
      
  </code></pre>
  <p>I want the full job description of each job and tried getting it using the job_description_tag, but it just returns 'N/A' (I put the if statement there for the case where a job posting does not have job description.).</p>
  <p>Each job post's output returns an 'N/A' so there is obviously something wrong.</p>
  <p>For reference, by inspecting the job description of one of the jobs, it is seen that the job_description_tag's html code I tried to scrape looks like this:</p>
  <pre class="lang-html prettyprint-override"><code>&lt;div id=&quot;jobDescriptionText&quot; class=&quot;jobsearch-jobDescriptionText&quot;&gt;&lt;ul&gt;
  &lt;li&gt;1+ years of experience as a Data Engineer or in a similar role&lt;/li&gt;
  &lt;li&gt;Experience with data modeling, data warehousing, and building ETL pipelines&lt;/li&gt;
  &lt;li&gt;Experience in SQL&lt;/li&gt;
  &lt;li&gt;Knowledge of python or any general purpose scripting language.&lt;/li&gt;
  &lt;li&gt;Experience with Big Data technologies such as Hive/Spark.&lt;/li&gt;
  &lt;/ul&gt;
  Mission Statement
  &lt;br&gt;The core mission of Amazon Web Services (AWS) Marketing is to educate customers about cloud computing and our services. Millions of customers engage with us every day across multiple channels. Imagine building a platform that enables AWS to speak to engineers, CTOs, CIOs, and CEOs, educate them about AWS services, and empower them on their journey to the cloud. Our services act as the foundation for announcing new AWS products and are uniquely positioned to redefine how our cloud community consumes information and engages with AWS.
  &lt;br&gt;&lt;br&gt;
  Overview
  &lt;br&gt;Would you like to support increasing customer base and the revenue for AWS, a market-leading cloud offering? Would you like to be part of a team focused on increasing awareness and adoption of the AWS platform by analyzing customer's behavior on and outside AWS websites? Do you want to empower our AWS marketing team make data-driven decisions that further establish AWS as leader in the cloud computing world?
  &lt;br&gt;&lt;br&gt;
  As a Data Engineer at AWS, you will be working in a large, extremely complex and dynamic data warehousing environment. We are looking for someone with the uncanny ability to integrate multiple heterogeneous data sources like Adobe Site Catalyst, Adobe Target, Sales Force, Adobe Connect with AWS central data warehouse and build efficient, flexible, and scalable data warehouse and reporting solutions. You should be enthusiastic about learning new technologies and be able to implement solutions using these technologies to enable upgrades of the existing platform. You should have excellent business and communication skills and be able to work with business owners to develop and define key business questions, then build the data sets that answer those questions. You should be expert at designing, implementing, and operating stable, scalable, low cost solutions to flow data from production systems into the data warehouse and into end-user facing reporting applications. Above all you should be passionate about working with huge data sets and someone who loves to bring datasets together to answer business questions and drive growth.
  &lt;br&gt;&lt;br&gt;
  At AWS, you have control over every layer you build. Instead of owning a small slice of an existing service, you will own a core segment of a growing marketing platform serving 1000s of internal customers and millions of external customers. You will build on multiple AWS services and have opportunities to engage directly with those teams to improve our core offerings. At AWS, we work with our customers on a daily basis to prove out our ideas, gather feedback, and improve the platform.
  &lt;br&gt;&lt;br&gt;
  &lt;b&gt;Location:&lt;/b&gt; This position must sit in Seattle, WA. Relocation assistance offered from within the US.
  &lt;br&gt;&lt;br&gt;
  &lt;ul&gt;
  &lt;li&gt;Graduate/Master degree in Computer Science, Engineering or related technical field.&lt;/li&gt;
  &lt;li&gt;Exceptional troubleshooting and problem-solving abilities.&lt;/li&gt;
  &lt;li&gt;Experience with Amazon Redshift or other distributed computing technology.&lt;/li&gt;
  &lt;li&gt;Industry experience as a Data Engineer or related specialty (e.g., Software Engineer, Business Intelligence Engineer, Data Scientist) with a track record of manipulating, processing, and extracting value from large datasets.&lt;/li&gt;
  &lt;li&gt;Experience with AWS Tools and Technologies.&lt;/li&gt;
  &lt;li&gt;Hands-on experience with cloud computing and UNIX/Linux based systems.&lt;/li&gt;
  &lt;li&gt;Demonstrated ability to work effectively across various internal organizations.&lt;/li&gt;
  &lt;li&gt;Excellent written and verbal communications skills.&lt;/li&gt;
  &lt;/ul&gt;
  Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us.&lt;/div&gt;
  </code></pre>
  <p>Any help will be appreciated!</p>