title: lxml - Is there any hacky way to keep &quot;?
body: <p>I noticed the xml entities <strong>&amp;quot</strong> will automatically force to convert to their real original characters:</p>
  
  <pre><code>&gt;&gt;&gt; from lxml import etree as et
  &gt;&gt;&gt; parser = et.XMLParser()
  &gt;&gt;&gt; xml = et.fromstring("&lt;root&gt;&lt;elem&gt;&amp;quot;hello world&amp;quot;&lt;/elem&gt;&lt;/root&gt;", parser)
  &gt;&gt;&gt; print et.tostring(xml, pretty_print=1)
  &lt;root&gt;
    &lt;elem&gt;"hello world"&lt;/elem&gt;
  &lt;/root&gt;
  
  &gt;&gt;&gt; 
  </code></pre>
  
  <p>I found one related old(2009-02-07) <a href="https://answers.launchpad.net/lxml/+question/60170" rel="nofollow">thread</a>:</p>
  
  <blockquote>
    <blockquote>
      <p>s = cStringIO.StringIO("""&quot;She&apos;s the MAN!&quot;""")
      e = etree.parse(s,etree.XMLParser(resolve_entities=False))</p>
    </blockquote>
    
    <p>Note that there's also etree.fromstring().</p>
    
    <blockquote>
      <p>etree.tostring(e)
      '"She\'s the MAN!"'</p>
      
      <p>I would have expected resolve_entities=False to have prevented the
      translation of, eg, &quot; to ".</p>
    </blockquote>
    
    <p>The "resolve_entities" option is meant for entities defined in a DTD
    of which you want to keep the reference instead of the resolved value.
    The entities you mention are part of the XML spec, not of a DTD.</p>
    
    <blockquote>
      <p>is there another way to prevent this behavior (or, if nothing else,
      reverse it after the fact)?</p>
    </blockquote>
    
    <p>Well, what you get is well-formed XML. May I ask why you need the
    entity references in the output?</p>
  </blockquote>
  
  <p>Still, the response is why you want to do that, there's no direct answer to this problem. I'm quite surprised because the etree parser force the conversion without giving an option to disable it.</p>
  
  <p>The following example shown why i need this solution, this xml is for <a href="http://kodi.wiki/view/Skinning" rel="nofollow">xbmc skinning</a> parser:</p>
  
  <pre><code>&gt;&gt;&gt; print open("/tmp/so.xml").read() #the original file
  &lt;window id="1234"&gt;
          &lt;defaultcontrol&gt;101&lt;/defaultcontrol&gt;
          &lt;controls&gt;
                  &lt;control type="button" id="101"&gt;
                          &lt;onfocus&gt;Dialog.Close(212)&lt;/onfocus&gt;
                          &lt;onfocus&gt;SetFocus(11)&lt;/onfocus&gt;
                  &lt;/control&gt;
                  &lt;control type="button" id="102"&gt;
                          &lt;visible&gt;StringCompare(VideoPlayer.PlotOutline,Stream.IsPlaying) + !Skin.HasSetting(Stream.IsUpdated)&lt;/visible&gt;
                          &lt;onfocus&gt;RunScript(script.test)&lt;/onfocus&gt;
                          &lt;onfocus&gt;SetFocus(11)&lt;/onfocus&gt;
                  &lt;/control&gt;
                  &lt;control type="button" id="103"&gt;
                          &lt;visible&gt;SubString(VideoPlayer.PlotOutline,Video.IsPlaying)&lt;/visible&gt;
                          &lt;onfocus&gt;Close&lt;/onfocus&gt;
                          &lt;onfocus&gt;RunScript(&amp;quot;/.xbmc/addons/script.hello.world/default.py&amp;quot;,&amp;quot;$INFO[VideoPlayer.Album]&amp;quot;,&amp;quot;$INFO[VideoPlayer.Genre]&amp;quot;)&lt;/onfocus&gt;
                  &lt;/control&gt;
          &lt;/controls&gt;
  &lt;/window&gt;
  
  &gt;&gt;&gt; root = et.parse("/tmp/so.xml", parser)
  &gt;&gt;&gt; r = root.getroot()
  &gt;&gt;&gt; for c in r:
  ...     for cc in c:
  ...         if cc.attrib.get('id') == "103":
  ...             cc.remove(cc[1]) #remove 1 element, it's just a demonstrate
  ... 
  &gt;&gt;&gt; o = open("/tmp/so.xml", "w")
  &gt;&gt;&gt; o.write(et.tostring(r, pretty_print=1)) #save it back
  &gt;&gt;&gt; o.close()
  &gt;&gt;&gt; print open("/tmp/so.xml").read() #the file after implemented 
  &lt;window id="1234"&gt;
          &lt;defaultcontrol&gt;101&lt;/defaultcontrol&gt;
          &lt;controls&gt;
                  &lt;control type="button" id="101"&gt;
                          &lt;onfocus&gt;Dialog.Close(212)&lt;/onfocus&gt;
                          &lt;onfocus&gt;SetFocus(11)&lt;/onfocus&gt;
                  &lt;/control&gt;
                  &lt;control type="button" id="102"&gt;
                          &lt;visible&gt;StringCompare(VideoPlayer.PlotOutline,Stream.IsPlaying) + !Skin.HasSetting(Stream.IsUpdated)&lt;/visible&gt;
                          &lt;onfocus&gt;RunScript(script.test)&lt;/onfocus&gt;
                          &lt;onfocus&gt;SetFocus(11)&lt;/onfocus&gt;
                  &lt;/control&gt;
                  &lt;control type="button" id="103"&gt;
                          &lt;visible&gt;SubString(VideoPlayer.PlotOutline,Video.IsPlaying)&lt;/visible&gt;
                          &lt;onfocus&gt;RunScript("/.xbmc/addons/script.hello.world/default.py","$INFO[VideoPlayer.Album]","$INFO[VideoPlayer.Genre]")&lt;/onfocus&gt;
                  &lt;/control&gt;
          &lt;/controls&gt;
  &lt;/window&gt;
  
  &gt;&gt;&gt; 
  </code></pre>
  
  <p>As you can see of the <strong>onfocus</strong> element under id "103" at the end, the <strong>&amp;quot</strong> are no longer in their original form, and it lead to bug if the "$INFO[VideoPlayer.Album]" variable contains nested quotes and become ""test"" which was invalid and error. </p>
  
  <p>So is it any hacky way i can keep <strong>&amp;quot</strong> in their original form ?</p>
  
  <p><strong>[UPDATE]:</strong>
  For someone who interest, the other 3 predefined xml entities, i.e. <strong>gt</strong>, <strong>lt</strong> and <strong>amp</strong> will only get converted by using <strong>method="html"</strong> and <strong>script</strong> tag. Either lxml VS xml.etree.ElementTree or python2 VS python3 have the same mechanism and make people confuse:</p>
  
  <pre><code>&gt;&gt;&gt; from lxml import etree as et
  &gt;&gt;&gt; r = et.fromstring("&lt;root&gt;&lt;script&gt;&amp;quot;&amp;apos;&amp;amp;&amp;gt;&amp;lt;&lt;/script&gt;&lt;p&gt;&amp;quot;&amp;apos;&amp;amp;&amp;gt;&amp;lt;&lt;/p&gt;&lt;/root&gt;")
  &gt;&gt;&gt; print et.tostring(r, pretty_print=1, method="xml")
  &lt;root&gt;
    &lt;script&gt;"'&amp;amp;&amp;gt;&amp;lt;&lt;/script&gt;
    &lt;p&gt;"'&amp;amp;&amp;gt;&amp;lt;&lt;/p&gt;
  &lt;/root&gt;
  
  &gt;&gt;&gt; print et.tostring(r, pretty_print=1, method="html")
  &lt;root&gt;&lt;script&gt;"'&amp;&gt;&lt;&lt;/script&gt;&lt;p&gt;"'&amp;amp;&amp;gt;&amp;lt;&lt;/p&gt;&lt;/root&gt;
  
  &gt;&gt;&gt; 
  </code></pre>
  
  <p><strong>[UPDATE2]:</strong>
  The following is the list of all possible html tags:</p>
  
  <pre><code>#https://github.com/html5lib/html5lib-python/blob/master/html5lib/sanitizer.py
  acceptable_elements = ['a', 'abbr', 'acronym', 'address', 'area',
  'article', 'aside', 'audio', 'b', 'big', 'blockquote', 'br', 'button',
  'canvas', 'caption', 'center', 'cite', 'code', 'col', 'colgroup',
  'command', 'datagrid', 'datalist', 'dd', 'del', 'details', 'dfn',
  'dialog', 'dir', 'div', 'dl', 'dt', 'em', 'event-source', 'fieldset',
  'figcaption', 'figure', 'footer', 'font', 'form', 'header', 'h1',
  'h2', 'h3', 'h4', 'h5', 'h6', 'hr', 'i', 'img', 'input', 'ins',
  'keygen', 'kbd', 'label', 'legend', 'li', 'm', 'map', 'menu', 'meter',
  'multicol', 'nav', 'nextid', 'ol', 'output', 'optgroup', 'option',
  'p', 'pre', 'progress', 'q', 's', 'samp', 'section', 'select',
  'small', 'sound', 'source', 'spacer', 'span', 'strike', 'strong',
  'sub', 'sup', 'table', 'tbody', 'td', 'textarea', 'time', 'tfoot',
  'th', 'thead', 'tr', 'tt', 'u', 'ul', 'var', 'video']
  from lxml import etree as et
  for e in acceptable_elements:
      r = et.fromstring(e.join(["&lt;", "&gt;hello&amp;amp;world&lt;/", "&gt;"]))
      s = et.tostring(r, pretty_print=1, method="html")
      closed_tag = "&lt;/" + e + "&gt;"
      if closed_tag not in s:
          print s
  </code></pre>
  
  <p>Run this code and you will see output as following:</p>
  
  <pre><code>&lt;area&gt;
  
  &lt;br&gt;
  
  &lt;col&gt;
  
  &lt;hr&gt;
  
  &lt;img&gt;
  
  &lt;input&gt;
  </code></pre>
  
  <p>As you can see, only opening tag printed and the rest was just go into black hole. I tested all 5 xml entities and all have the same behavior. It's so confusing. This did not happen when using HTMLParser, so i guess there's buggy between fromstring(method should be default to xml) and tostring(method="html") steps. And i found it has nothing to do with entities because "&lt; img >hello&lt; /img >"(without entities) is truncate into &lt; img > too(and hello just gone to nowhere, it can appear at anytime if use method="xml" to print out). </p>