How can I collect this data from a div using Selenium and Python
Question: I have been using Selenium and Pyton to scrape a webpage and I am having
difficulty collecting data that I want out of a div that has the following
structure:

    
    
    <div class="col span_6" style="margin-left: 12px;width: 47% !important;">
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Heading1</span>
        <span class="MainGridcolumn2">Text that I want</span>
      </div>
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Another heading</span>
        <span class="MainGridcolumn2">More text that I want</span>
      </div>
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Next heading</span>
        <span class="MainGridcolumn2">Even more text</span>
      </div>
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Yet another heading</span>
        <span class="MainGridcolumn2">Piece of text</span>
      </div>
    </div>
    

The div has a number of rows, each with 2 columns containing the data/text
inside of span tags. There are no CSS ids.

I'm only interested in collecting the text contained within the
'MainGridcolumn2' span classes.

I've tried the below to navigate to the first heading, with the intention of
then trying to use 'following_sibling' to move down to the next span tag
containing the text, but I can't even get this to work as it isn't returning
any text when I try to print it to the console:

    
    
    driver.find_element_by_xpath("//span['@class=MainGridcolumn1'][contains(text(), 'Heading1')]").text
    

and

    
    
    driver.find_element_by_xpath("//span[contains(text(), 'Heading1')]").text
    

Would appreciate some help!

Thanks


Answer: One way would be to get the the enclosing div i.e the grandparent and pull the
spans from that:

    
    
    h = """<div class="col span_6" style="margin-left: 12px;width: 47% !important;">
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Heading1</span>
        <span class="MainGridcolumn2">Text that I want</span>
      </div>
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Another heading</span>
        <span class="MainGridcolumn2">More text that I want</span>
      </div>
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Next heading</span>
        <span class="MainGridcolumn2">Even more text</span>
      </div>
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Yet another heading</span>
        <span class="MainGridcolumn2">Piece of text</span>
      </div>
    </div>
    
      <div class="MainGridRow">
        <span class="MainGridcolumn1">Yet another heading</span>
        <span class="MainGridcolumn2">Piece of text I don't want</span>
      </div>"""
    
    from lxml import html
    
    xm = html.fromstring(h)
    div = xm.xpath("//span[@class='MainGridcolumn1'][contains(text(), 'Heading1')]/../..")[0]
    print(div.xpath(".//span[@class='MainGridcolumn2']/text()"))
    

Which would give you:

    
    
    ['Text that I want', 'More text that I want', 'Even more text', 'Piece of text']
    

You could also just select the parent and get the parents siblings

    
    
    from lxml import html
    
    xm = html.fromstring(h)
    div = xm.xpath("//span[@class='MainGridcolumn1'][contains(text(), 'Heading1')]/..")[0]
    print(div.xpath(".//span[@class='MainGridcolumn2']/text() | .//following-sibling::div/span[@class='MainGridcolumn2']/text()"))
    

