Cannot seem to render XML or clean characters
Question: I'm trying to render my `sitemap.xml` using `flask`, however I cannot seem to
clean my input such that the rendering can work. The error is the following:

    
    
    error on line 23325 at column 83: PCDATA invalid Char value 11
    

The code I'm using is:

    
    
    url = prepend + "/explore/"+str(result['id'])+"/"+result['title'].encode('utf-8', 'xmlcharrefreplace').decode('utf-8')
    

I've also tried:

    
    
    url = prepend + "/explore/"+str(result['id'])+"/"+result['title'].encode('ascii', 'xmlcharrefreplace').decode('ascii')
    

Also removing what is Char 11, doesn't seem to work either.

    
    
    result["title"] = result["title"].replace('&#011;', '')
    

Is there a smarter way that I can clean any spurious characters or problem
solve more accurately?

I've tried to flag the problem. It looks like a linefeed or something along
those lines. I've tried before and after the URL is built.

    
    
    XX XXX XXX XXX  
                    Step by step guide
    

but, I cannot find anything in the python arsenal to remove the linefeed. I'm
trying, `replace('\r', '')`, `strip()` etc etc, but still this persists.


Answer: “Char value 11” (0xB in hex) refers to the vertical tab `\v`, **not** the
carriage return `\r`, **nor** the line feed `\n`.

The vertical tab is [not a valid character in an XML
document](https://www.w3.org/TR/xml/#charsets). It’s so invalid that you can’t
even represent it as a character reference like `&#011;`. You have to remove
it from the string yourself:

    
    
    result["title"] = result["title"].replace('\x0b', '')
    

While you’re at it, you may want to remove other invalid characters as well,
as they have a tendency to pop up inscrutably in user-supplied data and bite
you in production. I’m not aware of common library functions to do this, so I
came up with my own function that makes both XML and HTML5 more or less happy:

    
    
    import re
    
    def printable(s):
        # Based on `XML 1.0 section 2.2 <https://www.w3.org/TR/xml/#charsets>`_,
        # with the addition of U+0085,
        # which the W3C (Nu) validator also marked as a "forbidden code point".
        # Even with this code, the validator still complains about
        # "Text run is not in Unicode Normalization Form C"
        # and "Document uses the Unicode Private Use Area(s)".
        return re.sub(
            pattern=(u'[\u0000-\u0008\u000B\u000C\u000E-\u001F'
                     u'\u007F-\u009F\uD800-\uDFFF\uFDD0-\uFDEF\uFFFE\uFFFF]'),
            repl=u'\N{REPLACEMENT CHARACTER}',
            string=s
        )
    

As for `xmlcharrefreplace`, that’s not what you’re looking for at all. It’s
not an escaping mechanism for XML-invalid characters. It’s a way to preserve
characters that are outside of the chosen encoding. It would make sense in a
situation like this:

    
    
    >>> print(u'Liberté, égalité, fraternité!'.encode('ascii', 'xmlcharrefreplace'))
    Libert&#233;, &#233;galit&#233;, fraternit&#233;!
    

