Obtaining HTML source code with Python cookie
Question: 
        import urllib
    
        #my url here stored as url
    
        htmlfile = urllib.urlopen(url)
        htmltext = htmlfile.read()
        print(htmltext)
    

I'm trying to get source code from a url

I get source code but it is from a different page saying two things; please
enable cookies and this domain has banned your access based on your browser's
signature

Is there any way that anyone knows of to get the source code when the browser
knows your not actually on the page?


Answer: You may have to set an url opener

    
    
    def createOpener(self):
            handlers = []                                                       
            cj = MyCookieJar();
            cj.set_policy(cookielib.DefaultCookiePolicy(rfc2965=True))
            cjhdr = urllib2.HTTPCookieProcessor(cj)
            handlers.append(cjhdr)
            opener = urllib2.build_opener(*handlers)
            opener.addheaders = [('User-Agent', self.getUserAgent()),
                                      ('Host', 'google.com')]
            return opener
    

where the cookie jar is

    
    
    class MyCookieJar(cookielib.CookieJar):
        def _cookie_from_cookie_tuple(self, tup, request):
            name, value, standard, rest = tup
            version = standard.get('version', None)
            if version is not None:
                version = version.replace('"', '')
                standard["version"] = version
            return cookielib.CookieJar._cookie_from_cookie_tuple(self, tup, request)
    

At this point you create the opener and fetch the data reading the url handler
like:

    
    
    def fetchURL(self, url, data=None, headers={}):
            request = urllib2.Request(url, data, headers)
            self.opener = self.createOpener()
            urlHandle = self.opener.open(request)
            return urlHandle.read()
    

It's a good idea to have a `User-Agent` list and read from it:

    
    
    with open(ffpath) as f:
        USER_AGENTS_LIST = f.read().splitlines()
    

and get a random one from it

    
    
    index = random.randint(0,len(USER_AGENTS_LIST)-1)
    uA=USER_AGENTS_LIST[index]
    

To have a list of user agent take a look at
[here](http://stackoverflow.com/questions/36510058/javascript-user-agents-api-
to-fetch-and-filter-by-browser-class-type).

This is just to have and idea to do this without any external framework.

