Can't edit a URL with python
Question: I am new to python and just wanted to know if this is possible: I have scraped
a url using `urllib` and want to edit different pages.

**Example** : `http://test.com/All/0.html`

I want the `0.html` to become `50.html` and then `100.html` and so on ...


Answer: 
    found_url = 'http://test.com/All/0.html'
    
    base_url = 'http://test.com/All/'
    
    for page_number in range(0,1050,50):
        url_to_fetch = "{0}{1}.html".format(base_url,page_number)
    

That should give you URLs from `0.html` to `1000.html`

If you want to use `urlparse`(as suggested in comments to your question):

    
    
    import urlparse
    
    found_url = 'http://test.com/All/0.html'
    parsed_url = urlparse.urlparse(found_url)
    path_parts = parsed_url.path.split("/")
    
    for page_number in range(0,1050,50):
        new_path = "{0}/{1}.html".format("/".join(path_parts[:-1]), page_number)
        parsed_url = parsed_url._replace(path= new_path)
        print parsed_url.geturl()
    

Executing this script would give you the following:

    
    
    http://test.com/All/0.html
    http://test.com/All/50.html
    http://test.com/All/100.html
    http://test.com/All/150.html
    http://test.com/All/200.html
    http://test.com/All/250.html
    http://test.com/All/300.html
    http://test.com/All/350.html
    http://test.com/All/400.html
    http://test.com/All/450.html
    http://test.com/All/500.html
    http://test.com/All/550.html
    http://test.com/All/600.html
    http://test.com/All/650.html
    http://test.com/All/700.html
    http://test.com/All/750.html
    http://test.com/All/800.html
    http://test.com/All/850.html
    http://test.com/All/900.html
    http://test.com/All/950.html
    http://test.com/All/1000.html
    

Instead of printing in the for loop you can use the value of
parsed_url.geturl() as per your need. As mentioned, if you want to fetch the
content of the page you can use python `requests` module in the following
manner:

    
    
    import requests
    
    found_url = 'http://test.com/All/0.html'
    parsed_url = urlparse.urlparse(found_url)
    path_parts = parsed_url.path.split("/")
    
    for page_number in range(0,1050,50):
        new_path = "{0}/{1}.html".format("/".join(path_parts[:-1]), page_number)
        parsed_url = parsed_url._replace(path= new_path)
        # print parsed_url.geturl()
        url = parsed_url.geturl()
        try:
            r = requests.get(url)
            if r.status_code == 200:
                with open(str(page_number)+'.html', 'w') as f:
                    f.write(r.content)
        except Exception as e:
            print "Error scraping - " + url
            print e
    

This fetches the content from `http://test.com/All/0.html` till
`http://test.com/All/1000.html` and saves the content of each URL into its own
file. The file name on disk would be the file name in URL - `0.html` to
`1000.html`

Depending on the performance of the site you are trying to scrape from you
might experience considerable time delays in running the script. If
performance is of importance, you can consider using
[grequests](https://github.com/kennethreitz/grequests)

