Using Python to append a list of names to a single URL during a crawl of a URL
Question: I am looking to create a script that would crawl my URL and would append a
name to the end of the url during a search. For example
192.168.1.100/map/**foo**

I would then want to parse through the response and if the status code is 200
and the content-length is 83. I would want to output it to a text file. If
both these conditions are not matched then I will skip the printing of the
URL.

Here is my rough idea. I am looking for some general direction.

I would start with the URL I would either read an array or a list, depending
on the parameter length. I would then parse the response and look for the
conditions. If true I would write the URL to a text document else I would
continue the loop.

Thoughts? I'm not looking for you to write my code just to point me in the
general direction.

Thanks

Here is the latest revision based on @German Petrov code.

    
    
    import requests
    import urlparse
    
    url = "http://192.168.1.2/map/ShowPage.ashx?="
    names = ["admin","backup","contact","index","logs","news","reboot",
                  "register","test","users"]
    
    with open("/root/Desktop/urls.txt", 'a') as urls:
        for name in names:
           newUrl = url + name
           r = requests.get(newUrl)
           c = r.content
           if r.status_code == 200 and (("Unknown" in c) <>1):
                urls.write(url + '\n')
    

WOW!!! SUPER STUPID ERROR on my part. It helps if I add http:// to the url....
Also no need to do the urlparse.urljoin as it cuts off my full url.

This is why I like my super easy C# ide :-D


Answer: For the following, you have to install `requests`: <http://docs.python-
requests.org/en/latest/user/install/#install>

First

    
    
    import requests
    import urlparse
    
    baseurl= "192.168.1.2/map/ShowPage.ashx?="
    names = ["admin","backup","contact","index","logs","news","reboot",
                  "register","test","users"]
    
    for name in names:
        print urlparse.urljoin(baseurl, name)
    

gives you the following output:

    
    
    192.168.1.2/map/admin
    192.168.1.2/map/backup
    192.168.1.2/map/contact
    192.168.1.2/map/index
    192.168.1.2/map/logs
    192.168.1.2/map/news
    192.168.1.2/map/reboot
    192.168.1.2/map/register
    192.168.1.2/map/test
    192.168.1.2/map/users
    

Then you can update the code with get call:

    
    
    import requests
    import urlparse
    
    baseurl = "192.168.1.2/map/ShowPage.ashx?="
    names = ["admin","backup","contact","index","logs","news","reboot",
                  "register","test","users"]
    
    with open("C:\\urls.txt", 'a') as urls:    
        for name in names:
            url = urlparse.urljoin(baseurl, name)
            r = requests.get(url)
            if r.status_code == 200 and int(r.headers['content-length']) > 73: 
               urls.write(url) #Write it to the file
    

