beautifulsoup with get_text - handle spaces
Question: I using BS4 (python3) for extracting text from html file. My file looks like
this:

    
    
    <BODY>
    <P>Hello         World!</P>
    </BODY>
    </HTML>
    

When i calling `get_text()` method, the output is "Hello World!". Because it's
HTML, I was expected to get "Hello World!" (two or more spaces are replaced
with one space in HTML).

This Also relevant for this situation:

    
    
    <BODY>
    <P>Hello    
     World!</P>
    </BODY>
    </HTML>
    

I was expected to find "Hello World!" but it was "Hello \n World!".

How I can achieve my goal?


Answer: The problem is, neither `get_text(strip=True)` nor joining the
`.stripped_strings` would work here because there is a single
`NavigableString` in the `p` element in the second case and it's value is
`Hello\n World!`. The newline is inside the text node in other words.

In this case, you will have to _replace the newlines_ manually:

    
    
    soup.p.get_text().replace("\n", "")
    

* * *

Or, to also handle the `br` elements (replacing them with newlines), you can
make a converting function that will prepare the text for you:

    
    
    from bs4 import BeautifulSoup, NavigableString
    
    data = """
    <BODY>
    
    <P>Hello
     World!</P>
    
    <P>Hello
    <BR/>
     World!</P>
    
    </BODY>
    </HTML>
    """
    
    def replace_with_newlines(element):
        text = ''
        for elem in element.children:
            if isinstance(elem, NavigableString):
                text += elem.replace("\n", "").strip()
            elif elem.name == 'br':
                text += '\n'
        return text
    
    soup = BeautifulSoup(data, "html.parser")
    
    for p in soup.find_all("p"):
        print(replace_with_newlines(p))
    

Prints (no newlines in the first case, a single newline in the second):

    
    
    Hello World!
    Hello
    World!
    

