LXML escaped character conversion
Question: Ok so first of all I have a script which is operating on a dos file formatted
XML file. That is, the file has \r\n line terminations. Furthermore, the XML
file I am operating on has some newlines embedded within some attributes. The
XML editor which produced the XML encodes these newlines as: `&#xD;&#xA;`

I am using LXML and some of the processing I am doing changes these text
attributes into XML elements. The problem I am seeing is that the text blocks
with the newlines in them end up as elements but with some crud just before
the newline. I.e. `&#13;` which, by the way is equivalent to `&#xD;` as I
understand it.

Now, this, to me seems to be an issue in that the script I am executing is
executing within a linux environment and it dumps out a linux file formatted
file.

It appears to me as if LXML is correctly seeing that `&#xA` is an escaped
newline and changes this for an actual newline in the destination element. It
seems to be forgetting about the `&#xD` though.

I created a test xml file:

    
    
    <?xml version='1.0' encoding='UTF-8'?>
    <element1>
      <element2 value="0"/>
      <element3 documentation="Some documentation.&#xD;&#xA;&#xD;&#xA;Some more documentation"/>
    </element1>
    

And here is a sample python file to do some manipulation:

    
    
    #!/usr/bin/env python
    import re
    import argparse
    import sys
    
    import lxml.etree as ET
    xml = ET.parse('test.xml')
    root = xml.getroot()
    elem = root.find('element3')
    doc = ET.SubElement(elem, 'documenation')
    doc.text = elem.get('documentation')
    xml.write('out.xml', encoding='UTF-8', method="xml", pretty_print=True)
    

Here is the output:

    
    
     <element1>
      <element2 value="0"/>
      <element3 documentation="Some documentation.&#13;&#10;&#13;&#10;Some more documentation"><documenation>Some documentation.&#13;
    &#13;
    Some more documentation</documenation></element3>
    </element1>
    

I guess my question is this, is this how you guys would expect this to work? I
would have thought that in the documentation element, we would no longer have
the  references any more. It seems a bit inelegant if I have to now do some
grepping for these extraneous CR characters.


Answer: Yes, that is precisely how I'd expect it to work. While XML processors are
required to normalize line-endings on input, there is no requirement to
normalize line endings on output.

If I were in your shoes, I'd fix the code that generated that XML to avoid
putting carriage returns in attributes. Failing that, I'd try:

    
    
    doc.text = elem.get('documentation').replace('\r','')
    

