urllib2 returning nothing in python
Question: I am confused !!! can anybody tell me where the problem is??? this code used
to work properly but it started returning nothing since yesterday !! I did not
make any changes on it !!! does anybody have any idea???

    
    
    import re
    from re import sub
    import time
    import cookielib
    from cookielib import CookieJar
    import urllib2
    from urllib2 import urlopen
    import difflib
    import requests
    
    
    def twitParser():
    
            try:
                cj = CookieJar()            
                opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
                res=opener.open('https://twitter.com/haberturk')
                html=res.read()
    
                splitSource=re.findall(r'<p class="js-tweet-text tweet-text">(.*?)</p>',html)
                print len(splitSource)
    
                for item in splitSource:
                    aTweet = re.sub(r'<.*?>','',item)
                    print aTweet
    
                except Exception, e:
                    print str(e)
                    print 'ERROR IN MAIN TRY'
    
    
    
        twitParser()
    


Answer: If your code did not change, than propably something else did:

this tag does not exists anymore:

    
    
    <p class="js-tweet-text tweet-text">
    

Instead there is something like:

    
    
    ProfileTweet-text js-tweet-text u-dir
    

Although it is possible to get what you want using regexp, do not use it, use
a xml parser instead:

    
    
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html)
    ptags = soup.find_all("p")
    texts = [p.text for p in ptags if "js-tweet-text" in p["class"]]
    

Propably split up the function, first making sure you get the html, then if
you find p tags, then if you find any that meet your criteria.

As Wooble said, use the twitter api instead, these companies offer it so you
don't have to scrape and cost them resources.

