Detect URLs changes in python
Question: I need to detect urls changes when shops have a firesale on, I'm using
requests lib with no luck even when a firesale is on it still returns `No
deals on today` and check value is still `[u'http:', u'',
u'www.dealwebsite.co', u'Electroshop']`

primary shop url `http://www.dealwebsite.com/coolshop`

if firesale deals are on the primary shop url changes to this like a redirect
`http://www.dealwebsite.com/coolshop/firesale`

    
    
    import requests
    
    headers = {
    'User-Agent': 'Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/45.0.2454.101 Safari\/537.36'
    }
    
    select_shop = 'Electroshop'
    
    url = 'http://www.dealwebsite.co/' + select_shop
    r = requests.get(url, headers=headers, timeout=3)
    
    check = r.url.split('/')
    
    if len(check) != 5:
        print 'No deals on today'
        exit()
    else:
        print 'Firesale Deals on NOW!'
    


Answer: It seems that you can [track redirection](http://docs.python-
requests.org/en/master/user/quickstart/#redirection-and-history). For example:

    
    
    requests.get(url, headers=headers, timeout=3, allows_redirect=True)
    >>> r.url
    'url'
    
    >>> r.status_code
    200
    
    >>> r.history
    [<Response [301]>] # means that there was a redirect on the way
    

Actually, you could use just a HEAD request to verify the behavior - only if
you don't need to parse the result (as a HEAD response body is empty).

    
    
    >>> r = requests.head(url, headers=headers, timeout=3, allow_redirects=True)
    
    >>> r.url
    '..something...'
    
    >>> r.history
    [<Response [301]>] 
    

In theory, you could also prevent the redirect completely, and check for the
response status.

    
    
    >>> r = requests.get(url, headers=headers, timeout=3, allow_redirects=False)
    
    >>> r.status_code
    301
    
    >>> r.history
    []
    

Now, 301 might mean a redirect to firesale or somewhere else - you don't know.

**UPDATE 1**

An example with periscope.tv (it seems that the OP has issues with such a
website):

    
    
    >>> example = requests.get("https://periscope.tv/couchmode", allow_redirects=True)
    >>> example.status_code
    200
    >>> example.history
    [<Response [307]>]
    >>> example.history[0].url
    u'https://periscope.tv/couchmode'
    >>> example.url
    u'https://periscope.tv/w/aZwcYHNlcnZpY2V8MURYeHl6WUFaUWdLTerSfgniRKoRgIPbfxxlbAGofYQNBd8WZZTEelJ0KavI?mode=couch'
    

As you can see, example.history[0].url tells you what was the URL that
returned a 307 temporary redirect.

