Parsing XPath within non standard XML using lxml Python
Question: I’m trying to create a database of all patent information from Google Patents.
Much of my work so far has been using this very good answer from MattH in
[Python to parse non-standard XML file](http://stackoverflow.com/a/7336718).
My Python is too large to display so its linked
[here](http://textuploader.com/?p=6&id=CDEZq).

The source files [are
here](http://commondatastorage.googleapis.com/patents/grant_full_text/2010/ipg100105.zip):
a bunch of xml files appended together into one file with multiple headers.The
issue is trying to use the correct xpath expression when parsing this unsual
"non-standard" XML file which has multiple xml and dtd declarations. I have
been trying to use `"-".join(doc.xpath` to tie everything together when its
parsed out but the output creates blanks separated by hyphens for the
`<document-id>` and `<classification-national>` shown below

    
    
    <references-cited> <citation> 
    <patcit num="00001"> <document-id>
    <country>US</country> 
    <doc-number>534632</doc-number> 
    <kind>A</kind>
    <name>Coleman</name> 
    <date>18950200</date> 
    </document-id> </patcit>
    <category>cited by examiner</category>
    <classification-national><country>US</country>
    <main-classification>249127</main-classification></classification-national>
    </citation>
    

**Note** not all children exist within each `<citation>`, sometimes they are
not present at all.

How can I parse this xpath while trying to place hyphens between each data
entry for multiple entries under `<citation>` ?


Answer: From this XML (references.xml),

    
    
    <references-cited> 
      <citation> 
        <patcit num="00001"> 
          <document-id>
            <country>US</country> 
            <doc-number>534632</doc-number> 
            <kind>A</kind>
            <name>Coleman</name> 
            <date>18950200</date> 
          </document-id> 
        </patcit>
        <category>cited by examiner</category>
        <classification-national>
          <country>US</country>
          <main-classification>249127</main-classification>
        </classification-national>
      </citation>
    
      <citation>
        <patcit num="00002">
          <document-id>
            <country>US</country>
            <doc-number>D28957</doc-number>
            <kind>S</kind>
            <name>Simon</name>
            <date>18980600</date>
          </document-id>
        </patcit>
        <category>cited by other</category>
      </citation>
    </references-cited>
    

you can get the text content of every descendant of `<citation>` that has any
content as follows:

    
    
    from lxml import etree
    
    doc = etree.parse("references.xml")
    cits = doc.xpath('/references-cited/citation')
    
    for c in cits:
        descs = c.xpath('.//*')
        for d in descs:
            if d.text and d.text.strip():
                print "%s: %s"  %(d.tag, d.text)
        print
    

Output:

    
    
    country: US
    doc-number: 534632
    kind: A
    name: Coleman
    date: 18950200
    category: cited by examiner
    country: US
    main-classification: 249127
    
    country: US
    doc-number: D28957
    kind: S
    name: Simon
    date: 18980600
    category: cited by other
    

This variation:

    
    
    import sys
    from lxml import etree
    
    doc = etree.parse("references.xml")
    cits = doc.xpath('/references-cited/citation')
    
    for c in cits:
        descs = c.xpath('.//*')
        for d in descs:
            if d.text and d.text.strip():
                sys.stdout.write("-%s"  %(d.text))
        print
    

results in this output:

    
    
    -US-534632-A-Coleman-18950200-cited by examiner-US-249127
    -US-D28957-S-Simon-18980600-cited by other
    

