title: Python Selenium, Scraping LinkedIn: Looping through Work and Education Histories
body: <p>I am scraping data from LinkedIn profiles in Python using Selenium. It is mostly working but I can't figure out how to extract information for each employer or school in their history section.</p>
  <p>I am working from the following tutorial: <a href="https://www.linkedin.com/pulse/how-easy-scraping-data-from-linkedin-profiles-david-craven/" rel="nofollow noreferrer">https://www.linkedin.com/pulse/how-easy-scraping-data-from-linkedin-profiles-david-craven/</a></p>
  <p>And I am looking at this profile: <a href="https://www.linkedin.com/in/pauljgarner/?originalSubdomain=uk" rel="nofollow noreferrer">https://www.linkedin.com/in/pauljgarner/?originalSubdomain=uk</a></p>
  <p>Here is a partial snippet of the HTML section I am struggling with:</p>
  <pre><code>&lt;section id=&quot;experience-section&quot; class=&quot;pv-profile-section experience-section ember-view&quot;&gt;&lt;header class=&quot;pv-profile-section__card-header&quot;&gt;
    &lt;h2 class=&quot;pv-profile-section__card-heading&quot;&gt;
      Experience
    &lt;/h2&gt;
  
  &lt;!----&gt;&lt;/header&gt;
  
    &lt;ul class=&quot;pv-profile-section__section-info section-info pv-profile-section__section-info--has-more&quot;&gt;
  &lt;li id=&quot;ember136&quot; class=&quot;pv-entity__position-group-pager pv-profile-section__list-item ember-view&quot;&gt;        &lt;section id=&quot;1762786165&quot; class=&quot;pv-profile-section__card-item-v2 pv-profile-section pv-position-entity ember-view&quot;&gt;  &lt;div class=&quot;display-flex justify-space-between full-width&quot;&gt;
      &lt;div class=&quot;display-flex flex-column full-width&quot;&gt;
  &lt;a data-control-name=&quot;background_details_company&quot; href=&quot;/company/wagestream/&quot; id=&quot;ember138&quot; class=&quot;full-width ember-view&quot;&gt;          &lt;div class=&quot;pv-entity__logo company-logo&quot;&gt;
    &lt;img src=&quot;https://media-exp1.licdn.com/dms/image/C560BAQEkzVWoORqWFQ/company-logo_100_100/0/1615996325297?e=1631145600&amp;amp;v=beta&amp;amp;t=SoZQKV09PqqYxYTzbjqV4XTJa7HkGUZRe4QT0jU5hmE&quot; loading=&quot;lazy&quot; alt=&quot;Wagestream&quot; id=&quot;ember140&quot; class=&quot;pv-entity__logo-img EntityPhoto-square-5 lazy-image ember-view&quot;&gt;
  &lt;/div&gt;
  &lt;div class=&quot;pv-entity__summary-info pv-entity__summary-info--background-section &quot;&gt;
    &lt;h3 class=&quot;t-16 t-black t-bold&quot;&gt;Senior Software Engineer&lt;/h3&gt;
    &lt;p class=&quot;visually-hidden&quot;&gt;Company Name&lt;/p&gt;
    &lt;p class=&quot;pv-entity__secondary-title t-14 t-black t-normal&quot;&gt;
        Wagestream
          &lt;span class=&quot;pv-entity__secondary-title separator&quot;&gt;Full-time&lt;/span&gt;
    &lt;/p&gt;
      &lt;div class=&quot;display-flex&quot;&gt;
      &lt;h4 class=&quot;pv-entity__date-range t-14 t-black--light t-normal&quot;&gt;
        &lt;span class=&quot;visually-hidden&quot;&gt;Dates Employed&lt;/span&gt;
        &lt;span&gt;Apr 2021 – Present&lt;/span&gt;
      &lt;/h4&gt;
        &lt;h4 class=&quot;t-14 t-black--light t-normal&quot;&gt;
          &lt;span class=&quot;visually-hidden&quot;&gt;Employment Duration&lt;/span&gt;
          &lt;span class=&quot;pv-entity__bullet-item-v2&quot;&gt;3 mos&lt;/span&gt;
        &lt;/h4&gt;
    &lt;/div&gt;
  
    &lt;h4 class=&quot;pv-entity__location t-14 t-black--light t-normal block&quot;&gt;
      &lt;span class=&quot;visually-hidden&quot;&gt;Location&lt;/span&gt;
      &lt;span&gt;London, England, United Kingdom&lt;/span&gt;
    &lt;/h4&gt;
  &lt;!----&gt;
  &lt;/div&gt;
  
  &lt;/a&gt;
  &lt;!----&gt;    &lt;/div&gt;
  
  &lt;!----&gt;  &lt;/div&gt;
  &lt;/section&gt;
  
  </code></pre>
  <p>And this is followed by more &quot;li&quot; sections. So the overall history section can be identified with id=&quot;experience-section&quot;, work (as opposed to education) history can be identified in the &quot;ul&quot; section class=&quot;pv-profile-section__section-info section-info pv-profile-section__section-info--has-more&quot;. The information for the first job in the list can be identified with &quot;li&quot; section id=&quot;ember136&quot;.</p>
  <p>I am trying to get job title, company, years in job etc. from this section but can't figure out how to do it. Here is a bit of python code to show what I have tried (skipping my log-in):</p>
  <pre><code>from parsel import Selector
  from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  import requests
  
  path = r'C:\Program Files (x86)\chromedriver_win32\chromedriver.exe'
  driver = webdriver.Chrome(path)
  
  # driver.get method() will navigate to a page given by the URL address
  driver.get('https://www.linkedin.com/in/pauljgarner/?originalSubdomain=uk')
  
  text=driver.page_source
  sel = Selector(text) 
  
  # Using the &quot;Copy xPath&quot; option in Inspect in Google Chrome, I can manually extract the company name
  sel.xpath('//*[@id=&quot;ember187&quot;]/div[2]/p[2]/text()').extract_first()  
  
  # This will give me all of the text in the Work Experience section
  stuff = driver.find_element_by_id(&quot;experience-section&quot;)
  items = html_list.find_elements_by_tag_name(&quot;ul&quot;)
  items = html_list.find_elements_by_tag_name(&quot;h3&quot;)
  for item in items:
      print(type(item))
      text = item.text
      print(text)
  
  </code></pre>
  <p>But these approaches are not great for an automated and systematic extraction of info from each job across profiles. What I would like to do is something like looping across &quot;li&quot; sections within each &quot;ul&quot; section, and within the &quot;li&quot; part, extract only the company name with class = &quot;pv-entity__secondary-title t-14 t-black t-normal&quot;. But find_element_by_class_name only yields NoneTypes.</p>
  <p>I'm not sure conceptually how to generate an iterable list of &quot;ul&quot; and &quot;li&quot; with selenium, and within each iteration extract specific bits of text using class names.</p>