How can I extract the text outside the <em> tag in BeautifulSoup
Question: Can someone help me extract the test that is after the _From_ , I want to
extract the sender name. It is situated right outside the em tag. I'm using
the python BeautifulSoup package.

Here is a link to the webpage: <http://seclists.org/fulldisclosure/2016/Jan/0>

I was able to extract the email title successfully since is was in a tag.
There are no other div's or classes in the html page.

This is the html code of the page:

Here is what I've tried

    
    
    def title_spider(max_pages):
        page = 0
        while page <= max_pages:
            url = 'http://seclists.org/fulldisclosure/2016/Jan/' + str(page)
            source_code = requests.get(url)
            plain_text = source_code.text
            soup = BeautifulSoup(plain_text, "html.parser")
            for email_title in soup.find('b'):
                title = email_title.string
                print(title)
    
            for date_stamp in soup.em:
                date = date_stamp
                print(date)
            page += 1
    
    title_spider(2)
    

`


Answer: You want the next sibling and if you want the specific em's From and Date you
can combine with a regex:

    
    
    import re
    
    def title_spider(max_pages):
        for page in range(max_pages + 1):
            url = 'http://seclists.org/fulldisclosure/2016/Jan/{}'.format(page) 
            source_code = requests.get(url)
            plain_text = source_code.text
            soup = BeautifulSoup(plain_text, "html.parser")
            for email_title in soup.find('b'):
                title = email_title.string
                print(title)
    
            for em in soup.find_all("em", text=re.compile("From|Date")):
                print(em.text, em.next_sibling)
    

Which gives you:

    
    
    In [5]: title_spider(2)
    Alcatel Lucent Home Device Manager - Management Console Multiple XSS
    From : Uğur Cihan KOÇ <u.cihan.koc () gmail com>
    Date : Sun, 3 Jan 2016 13:20:53 +0200
    Executable installers/self-extractors are vulnerable^WEVIL  (case 17): Kaspersky Labs utilities
    From : "Stefan Kanthak" <stefan.kanthak () nexgo de>
    Date : Sun, 3 Jan 2016 16:12:50 +0100
    Possible vulnerability in F5 BIG-IP LTM - Improper input validation of the HTTP version number of the HTTP reqest allows any payload size and conent to pass through
    From : Eitan Caspi <eitanc () yahoo com>
    Date : Sun, 3 Jan 2016 21:10:27 +0000 (UTC)
    

