get specific text such as "Something new" using BeautifulSoup Python
Question: I am making a focused crawler and facing an issue while find a for a key
phrase in the document.

Supposing the key phrase I want to search in the document is "Something new"

using BeautifulSoup with python I do the following

    
    
    if soup.find_all(text = re.compile("Something new",re.IGNORECASE)):
          print true
    

I want it to print true only for the following cases

"something new" --> true

"$#something new,." --> true

AND not for the following cases:

"thisSomething news" --> false

"Somethingnew" --> false

assuming special characters are allowed.

Has anyone ever done something like this before. ??

Thanks for the help.


Answer: Then, search for `something new` and don't apply `re.IGNORECASE`:

    
    
    import re
    
    from bs4 import BeautifulSoup
    
    
    data = """
    <div>
        <span>something new</span>
        <span>$#something new,.</span>
        <span>thisSomething news</span>
        <span>Somethingnew</span>
    </div>
    """
    
    soup = BeautifulSoup(data)
    for item in soup.find_all(text=re.compile("something new")):
        print item
    

Prints:

    
    
    something new
    $#something new,.
    

* * *

You can also take a non-regex approach and [pass a
function](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#a-function)
instead of a compiled regex pattern:

    
    
    for item in soup.find_all(text=lambda x: 'something new' in x):
        print item
    

For the example HTML used above, it also prints:

    
    
    something new
    $#something new,.
    

