title: Prettify with BeautifulSoup using a formatter that will preserve &nbsp AND Cyrillic characters?
body: <p>I'm generating some HTML with python and BeautifulSoup4. At the end, I'd like to prettify the generated HTML. If I prettify as follows:</p>
  <pre><code>soup.prettify()
  </code></pre>
  <p>BeautifulSoup converts all the &amp;nbsp characters to spaces. Unfortunately, my webpage relies on having these &amp;nbsp characters. After some guidance, I realized that this can be overcome by supplying a formatter to prettify:</p>
  <pre><code>soup.prettify(formatter='html')
  </code></pre>
  <p>Unfortunately, when I do this, though the &amp;nbsp characters are preserved, BeautifulSoup encodes the Cyrillic (Russian) characters in my HTML, making them unreadable to me. This leaves the formatter='html' option off limits to me.</p>
  <p>(<code>formatter='minimal'</code> and <code>formatter=None</code> also don't work; they leave Cyrillic alone, but take away the &amp;nbsp.)</p>
  <p>After looking at the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#output-formatters" rel="nofollow noreferrer">BeautifulSoup docs</a>, I realized you can specify your own custom formatter using BeautifulSoup's Formatter class. Unfortunately, I am unsure how this class works. I have tried to find documentation for the Formatter class but I am unable. Does anyone know if it's possible to write a custom formatter, that will tell BeautifulSoup to preserve &amp;nbsp characters (and leave my Cyrillic characters alone)? Or, is there any documentation for how this class works exactly? There are some examples in that section of the BS documentation, but after reading them, I am still unclear how to accomplish what I'm trying to accomplish.</p>
  <p><strong>EDIT: I have found <a href="https://tedboy.github.io/bs4_doc/8_output.html" rel="nofollow noreferrer">different documentation</a>, which makes it much clearer. The custom formatter is just a function you pass to the 'formatter' arg (i.e. <code>prettify(formatter=my_func)</code>, where my_func is a function you define on your own); it gets called once for every String and attribute value encountered, passing that value to the function and using whatever the function returns as the output in prettify. I have experimented writing my own formatter function, and I'm able to detect if an &amp;nbsp is there, but unsure what to return from the function, so that prettify will output the &amp;nbsp. See 'Example 3' below for my dummy formatter to detect &amp;nsbp.</strong></p>
  <p>Here is a dummy example demonstrating the problem:</p>
  <p><em><strong>EXAMPLE 1: Using prettify without a formatter</strong></em></p>
  <pre><code>from bs4 import BeautifulSoup
  hello = '&lt;span&gt;Привет,&amp;nbspмир&lt;/span&gt;'
  soup = BeautifulSoup(hello, 'html.parser')
  print(&quot;\nBefore prettify:\n{}&quot;.format(soup))
  soup = soup.prettify()
  print(&quot;\nAfter prettify:\n{}&quot;.format(soup))
  </code></pre>
  <p><em>Output - Cyrillic characters are fine, but &amp;nbsp are converted to ws</em></p>
  <pre><code>Before prettify:
  &lt;span&gt;Привет, мир&lt;/span&gt;
  
  After prettify:
  &lt;span&gt;
   Привет, мир
  &lt;/span&gt;
  </code></pre>
  <p><em><strong>EXAMPLE 2: Using prettify with formatter='html'</strong></em></p>
  <pre><code>from bs4 import BeautifulSoup
  hello = '&lt;span&gt;Привет,&amp;nbspмир&lt;/span&gt;'
  soup = BeautifulSoup(hello, 'html.parser')
  print(&quot;\nBefore prettify:\n{}&quot;.format(soup))
  soup = soup.prettify(formatter='html')
  print(&quot;\nAfter prettify:\n{}&quot;.format(soup))
  </code></pre>
  <p><em>output: &amp;nbsp are preserved, but Cyrillic characters get converted unreadable</em></p>
  <pre><code>Before prettify:
  &lt;span&gt;Привет, мир&lt;/span&gt;
  
  After prettify:
  &lt;span&gt;
   &amp;Pcy;&amp;rcy;&amp;icy;&amp;vcy;&amp;iecy;&amp;tcy;,&amp;nbsp;&amp;mcy;&amp;icy;&amp;rcy;
  &lt;/span&gt;
  </code></pre>
  <p><em><strong>Example 3: Supplying a custom formatter.</strong> This is just a dummy formatter for the sake of the example, to detect if &amp;nbsp is there.  What should I return from this function, if I want &amp;nbsp to be preserved? (p.s., it seems <a href="https://stackoverflow.com/questions/66895175/beautifulsoup-find-tag-with-text-containing-nbsp">&amp;nbsp are parsed as \xa0</a>, which is why I'm checking for it this way)</em></p>
  <pre><code>def check_for_nbsp(str):
      if '\xa0' in str:
          return str+&quot; &lt;-- HAS&quot;
      else:
          return str+&quot; &lt;-- DOESN'T HAVE&quot;
  
  hello = '&lt;span&gt;Привет,&amp;nbspмир&lt;/span&gt;'
  soup = BeautifulSoup(hello, 'html.parser')
  print(&quot;\nBefore prettify:\n{}&quot;.format(soup))
  soup = soup.prettify(formatter=check_for_nbsp)
  print(&quot;\nAfter prettify:\n{}&quot;.format(soup))
  </code></pre>
  <p>Output:</p>
  <pre><code>Before prettify:
  &lt;span&gt;Привет, мир&lt;/span&gt;
  
  After prettify:
  &lt;span&gt;
   Привет, мир &lt;-- HAS
  &lt;/span&gt;
  </code></pre>
  <p><strong>Is there a way to get the best of both worlds - preserve the &amp;nbsp AND the Cyrillic characters? Alternatively, is there a realiable python package that prettifies HTML other than BeautifulSoup?</strong></p>
  <p><a href="https://stackoverflow.com/questions/69782625/beautifulsoup-prettify-encoding-non-english-cyrillic-characters-strangely">Here is a previous Stackoverflow question</a> I posted regarding mangling the Cyrillic characters - that's what led me to understand I should remove the formatter='html' option, unfortunately this removes the &amp;nbsp characters, which is equally as problematic.</p>