Python lxml's XPath not finding <ul> in <p> tags
Question: I have a problem with the XPath function of pythons lxml. A minimal example is
the following python code:

    
    
    from lxml import html, etree
    
    text = """
          <p class="goal">
                <strong>Goal</strong> <br />
                <ul><li>test</li></ul>
            </p>
    """
    
    tree = html.fromstring(text)
    thesis_goal = tree.xpath('//p[@class="goal"]')[0]
    print etree.tostring(thesis_goal)
    

Running the code produces

    
    
    <p class="goal">
                <strong>Goal</strong> <br/>
                </p>
    

As you can see, the entire `<ul>` block is lost. This also means that it is
not possible to address the `<ul>` with an XPath along the lines of
`//p[@class="goal"]/ul`, as the `<ul>` is not counted as a child of the `<p>`.

Is this a bug or a feature of lxml, and if it is the latter, how can I get
access to the entire contents of the `<p>`? The thing is embedded in a larger
website, and it is not guaranteed that there will even _be_ a `<ul>` tag
(there may be another `<p>` inside, or anything else, for that matter).

**Update** : Updated title after answer was received to make finding this
question easier for people with the same problem.


Answer: `ul` elements (or more generally [flow
content](http://www.w3.org/TR/html5/dom.html#flow-content-1)) are [not allowed
inside `p` elements](http://stackoverflow.com/q/5681481/190597) (which can
only contain [phrasing content](http://www.w3.org/TR/html5/dom.html#phrasing-
content-1)). Therefore `lxml.html` parses `text` as

    
    
    In [45]: print(html.tostring(tree))
    <div><p class="goal">
                <strong>Goal</strong> <br>
                </p><ul><li>test</li></ul>
    
    </div>
    

The `ul` follows the `p` element. So you could find the `ul` element using the
XPath

    
    
    In [47]: print(html.tostring(tree.xpath('//p[@class="goal"]/following::ul')[0]))
    <ul><li>test</li></ul>
    

