Scraping names of directors from a website
DRY

Reduce duplicated logic using helper functions.
Currently you have 2 lines of code for each field you extract,
for example:


companys = sel.xpath('//*[@id="company-name"]/text()').extract()
companys = [company.strip() for company in companys]

names = sel.xpath('//*[@id="%s"]/a/text()' % string1).extract()
names = [name.strip() for name in names]



This is tedious. You could capture the common logic in a helper function, for example:

def to_list(xpath):
    return [v.strip() for v in xpath.extract()]


With that, much of the code can be simplified:

companys = to_list(sel.xpath('//*[@id="company-name"]/text()'))
names = to_list(sel.xpath('//*[@id="%s"]/a/text()' % string1).extract())


Repeated operations

Here, an xpath lookup is performed twice:


if sel.xpath('//*[@id="%s"]/text()' % string3):
    dateofbirths = sel.xpath('//*[@id="%s"]/text()' % string3).extract()
else:
    dateofbirths = ["n/a"]
dateofbirths = [dateofbirth.strip() for dateofbirth in dateofbirths]



It would be better to avoid that:

dateofbirths = to_list(sel.xpath('//*[@id="%s"]/text()' % string3))
if not dateofbirths:
    dateofbirths = ["n/a"]


Use "...".format(...)

The "%s" % ... style formatting is old, it's recommended to use the format function instead, for example:

names = sel.xpath('//*[@id="{}"]/a/text()'.format(string1)).extract()


Formatting

Python has a style guide called PEP8, I suggest to follow it.