Python, selenium and catching specific text in div outside any span or something
Question: I came across a page i want to scrap a bit - and i cried a lot looking at the
structure of address details section. But let's be specific:

I have a result structure like this:

    
    
    <div class="A">
      <div class="B">
        <div class="INFO">
          Foo Bar School of Baz and Qux
          <br>
          <span class="TYPE">
            Wibble school of Wobble
          </span>
          <br>
          <br>
          12th Hurr Durr Street, 12345 Derp
          <br>
          <span>Phone: 123 567 890 </span> <br>
          <span>Fax: 666 69 69 69 </span>
          <br>
        </div>
      </div>
    </div>
    

and i want to extract the name and adress of the place using selenium in
python. So i wrote xpath which happen to work:

    
    
    (//div[@class='INFO'])[1]//text()[not(parent::span) and normalize-space()]
    

But since things i want to extract aren't elements, just text, they are
specified with text() with "don't be inside span" and "don't be whitespace".

    
    
    driver.find_element_by_xpath(thing_i_wrote_above)
    

throws

    
    
    mon.exceptions.InvalidSelectorException: Message: The given selector <the same xpath> is: [object Text]. It should be an element.
    

I don't see any way to select the element, because closest one is INFO, which
happens to contain all of the informations. How to grab these things?


Answer: You could get the children text nodes with a piece of JavaScript:

    
    
    # get the container
    element = driver.find_element_by_css_selector(".INFO")
    
    # return an array with the text from the children text nodes
    texts = driver.execute_script("""
      return Array.from(arguments[0].childNodes)
        .filter(function(o){return o.nodeType === 3 && o.nodeValue.trim().length;})
        .map(function(o){return o.nodeValue.trim();})
      """, element)
    
    print texts
    

You could also use BeautifulSoup to parse the html from the container:

    
    
    from bs4 import BeautifulSoup
    
    # get the container
    element = driver.find_element_by_css_selector(".INFO")
    
    # parse the HTML from the container
    bs = BeautifulSoup(element.get_attribute("outerHTML"))
    
    # list all the children text nodes
    texts = [v.strip() for v in bs.html.body.div.findAll(text=True, recursive=False) if v.strip()]
    
    print texts
    

