Writing and saving scraped documents in several sheets in an Excel file I've written some code in Python for the purpose of scraping names and URLs from several links found in the left sided bar in a webpage and populate the data in several sheets (also giving each sheet a new name taking a customized portion from URL) in an Excel workbook so that things do not get messy and the data can be located separately. I tried to do the whole thing accurately and found it working great. It would be very nice if I get any better idea other than what I did here.

Here is what I tried with:

import requests
from lxml import html
from pyexcel_ods3 import save_data

web_link = "http://www.wiseowl.co.uk/videos/"
main_url = "http://www.wiseowl.co.uk"

def get_links(page):

    response = requests.Session().get(page)
    tree = html.fromstring(response.text)
    data = {}
    titles = tree.xpath("//ul[@class='woMenuList']//li[@class='woMenuItem']/a/@href")
    for title in titles:
        if "author" not in title and "year" not in title:
            get_docs(data, main_url + title)

def get_docs(data, url):

    response = requests.Session().get(url)
    tree = html.fromstring(response.text)
    for item in tree.xpath("//p[@class='woVideoListDefaultSeriesTitle']"):
        title = item.findtext('.//a')
        link = item.xpath('.//a/@href')[0]
        sheetname = url.split("videos/")[1].split("/")[1].split(".")[0] 
        if sheetname in data:  
            data[sheetname].append([title, str(link)])  
        else:  
            data[sheetname] = [[title, str(link)]]
    save_data("documents.ods", data)

if __name__ == '__main__':
    get_links(web_link)
Overall I don't think it's terrible. Suggestions:


Add a shebang to the top of your script so that it's executable in Posix-like environments: #!/usr/bin/env python
You create a session every time you call get_docs. Don't do this unless absolutely necessary; instead, pass the session created in get_links to be reused.
Put this through a PEP8 linter, such as that built into PyCharm, for minor formatting suggestions, such as there being two newlines before every top-level function.
Your sheetname (which should be called sheet_name by Python standard) smells like it should be a regex instead of a series of splits.
Condense your if ... in ... else to a setdefault(); something like:


data.setdefault(sheet_name, []).append([title, str(link)])