Need to dump entire DOM tree with element id from selenium server
Question: I have been using python selenium for web automation testing. The key part of
automation is to find the right element for a user-visible object in a HTML
page. The following API will work most of the time, but not all the time.

    
    
    find_element_by_xxx,  xxx can be id, name, xpath, tag_name etc. 
    

When HTML page is too complicated, I would like to search the dom tree. Wonder
if it's possible to ask the selenium server to serialize the entire DOM (with
the element id that can be used to perform action on through webdriver
server). Client side (python script) can do its own search algorithm to find
the right element.

Note that python selenium can get the entire html page by

    
    
    drv.page_source
    

However, parsing this doesn't give the internal element id from selenium
server's point of view, hence not useful.

**EDIT1:** Paraphrase it to make it more clear (thanks @alecxe): what's needed
here is a serialized representation of all the DOM elements (with their DOM
structure preserved) in the selenium server, this serialized representation
can be sent to the client side (a python selenium test app) which can do its
own search.


Answer: ### The Problem

Ok, so there may be cases where you need to perform some substantial
processing of a page on the client (Python) side rather than on the server
(browser) side. For instance, if you have some sort of machine learning system
already written in Python and it needs to analyze the whole page before
performing actions on them, then although it is possible to do it with a bunch
of `find_element` calls, this gets very expensive because each call is a
round-trip between the client and the server. And rewriting it to work in the
browser may be too expensive.

### Why Selenium's Identifiers wont' do it

However, I do not see an _efficient_ way to get a serialization of the DOM
_together_ with Selenium's own identifiers. Selenium creates these identifiers
on an as-needed basis, when you call `find_element` or when DOM nodes are
returned from an `execute_script` call (or passed to the callback that
`execute_async_script` gives to the script). But if you call `find_element` to
get identifiers for each element, then you are back to square one. I could
imagine decorating the DOM in the browser with the required information but
there is no public API to request some sort of pre-assignment of `WebElement`
ids. As a matter of fact, these identifiers are designed to be opaque so even
if a solution managed somehow to get the required information, I'd be
concerned about cross-browser viability and ongoing support.

### A Solution

There is however a way to get an addressing system that would work on both
sides: XPath. The idea is to parse the DOM serialization into a tree on the
client side and then get the XPath of the nodes you are interested in and use
this to get the corresponding WebElement. So if you'd have to perform dozens
of client-server roundtrips to determine which single element you need to
perform a click on, you'd be able so reduce this to an initial query of the
page source plus a single `find_element` call with the XPath you need.

Here is a super simple proof of concept. It fetches the main input field of
the Google front page.

    
    
    from StringIO import StringIO
    
    from selenium import webdriver
    import lxml.etree
    
    #
    # Make sure that your chromedriver is in your PATH, and use the following line...
    #
    driver = webdriver.Chrome()
    #
    # ... or, you can put the path inside the call like this:
    # driver = webdriver.Chrome("/path/to/chromedriver")
    #
    
    parser = lxml.etree.HTMLParser()
    
    driver.get("http://google.com")
    
    # We get this element only for the sake of illustration, for the tests later.
    input_from_find = driver.find_element_by_id("gbqfq")
    input_from_find.send_keys("foo")
    
    html = driver.execute_script("return document.documentElement.outerHTML")
    tree = lxml.etree.parse(StringIO(html), parser)
    
    # Find our element in the tree.
    field = tree.find("//*[@id='gbqfq']")
    # Get the XPath that will uniquely select it.
    path = tree.getpath(field)
    
    # Use the XPath to get the element from the browser.
    input_from_xpath = driver.find_element_by_xpath(path)
    
    print "Equal?", input_from_xpath == input_from_find
    # In JavaScript we would not call ``getAttribute`` but Selenium treats
    # a query on the ``value`` attribute as special, so this works.
    print "Value:", input_from_xpath.get_attribute("value")
    
    driver.quit()
    

Notes:

  1. The code above does not use `driver.page_source` because Selenium's documentation states that there is no guarantee as to the freshness of what it returns. It could be the state of the current DOM or the state of the DOM when the page was first loaded.

  2. This solution suffers from the exact same problems that `find_element` suffers from regarding dynamic contents. If the DOM changes while the analysis is occurring, then you are working on a stale representation of the DOM.

  3. If you have to generate JavaScript events while performing the analysis, and these events change the DOM, then you'd need fetch the DOM again. (This is similar to the previous point but a solution that uses `find_element` calls could conceivably avoid the problem I'm talking about in _this_ point by ordering the sequence of calls carefully.)

  4. `lxml`'s tree could possibly differ _structurally_ from the DOM tree in such a way that the XPath obtained from `lxml` does not address the corresponding element in the DOM. What `lxml` processes is the cleaned up serialized view that the browser has of the HTML passed to it. Therefore, **so long as the code is written to prevent the problems I've mentioned in point 2 and 3** , I do not see this as a likely scenario, but it is not impossible.

