We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
SD failing since at least 2018-07-25
SD has been failing since 2018-07-25

Based on automated runs it appears that SD has not run successfully in 2 days (2018-07-25).


```
  02:01:45 INFO pupa: save post 9 as post_c1c5c0f0-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 10 as post_c1c5c244-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 11 as post_c1c5c38e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 12 as post_c1c5c4d8-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 13 as post_c1c5c622-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 14 as post_c1c5c76c-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 15 as post_c1c5c8ac-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 16 as post_c1c5c9ec-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 17 as post_c1c5cb36-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 18 as post_c1c5cc76-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 19 as post_c1c5cdc0-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 20 as post_c1c5cf00-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 21 as post_c1c5d04a-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 22 as post_c1c5d18a-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 23 as post_c1c5d2d4-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 24 as post_c1c5d41e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 25 as post_c1c5d55e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 27 as post_c1c5d69e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 29 as post_c1c5d7e8-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 30 as post_c1c5d9f0-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 31 as post_c1c5dbb2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 32 as post_c1c5dcf2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 33 as post_c1c5de5a-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 34 as post_c1c5dfc2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 35 as post_c1c5e13e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 26A as post_c1c5e292-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 26B as post_c1c5e3d2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 28A as post_c1c5e508-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 28B as post_c1c5e648-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx
02:01:45 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/default.aspx
02:01:46 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/default.aspx
loaded Open States pupa settings...
sd (scrape, import)
  people: {}
  bills: {}
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 120, in do_scrape
    raise ScrapeError('no objects returned from {} scrape'.format(self.__class__.__name__))
pupa.exceptions.ScrapeError: no objects returned from SDLegislatorScraper scrape
```

Visit http://bobsled.openstates.org for more info.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `openstates/sd/people.py`
Content:
```
1 import re
2 
3 from pupa.scrape import Person, Scraper, Organization
4 
5 import lxml.html
6 
7 
8 class SDLegislatorScraper(Scraper):
9 
10     def scrape(self, chambers=None):
11         self._committees = {}
12 
13         # emails are on the contact page, fetch once and the
14         # legislator scrapers can find their emails there
15         contact_page_url = \
16             'http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx'
17         contact_page = self.get(contact_page_url).text
18         contact_page = lxml.html.fromstring(contact_page)
19 
20         url = 'http://www.sdlegislature.gov/Legislators/default.aspx'
21         if chambers is None:
22             chambers = ['upper', 'lower']
23         for chamber in chambers:
24             if chamber == 'upper':
25                 search = 'Senate Legislators'
26             else:
27                 search = 'House Legislators'
28 
29             page = self.get(url).text
30             page = lxml.html.fromstring(page)
31             page.make_links_absolute(url)
32 
33             # Legisltor listing has initially-hidden <div>s that
34             # contain the members for just a particular chamber
35             for link in page.xpath(
36                 "//h4[text()='{}']".format(search) +
37                 "/../span/section/table/tbody/tr/td/a"
38             ):
39                 name = link.text.strip()
40                 yield from self.scrape_legislator(name, chamber, link.attrib['href'], contact_page)
41         yield from self._committees.values()
42 
43     def scrape_legislator(self, name, chamber, url, contact_page):
44         page = self.get(url).text
45         page = lxml.html.fromstring(page)
46         page.make_links_absolute(url)
47 
48         party = page.xpath("string(//span[contains(@id, 'Party')])")
49         party = party.strip()
50 
51         if party == 'Democrat':
52             party = 'Democratic'
53 
54         district = page.xpath("string(//span[contains(@id, 'District')])")
55         district = district.strip().lstrip('0')
56 
57         occupation = page.xpath(
58             "string(//span[contains(@id, 'Occupation')])")
59         occupation = occupation.strip()
60 
61         (photo_url, ) = page.xpath('//img[contains(@id, "_imgMember")]/@src')
62 
63         office_phone = page.xpath(
64             "string(//span[contains(@id, 'CapitolPhone')])").strip()
65 
66         legislator = Person(primary_org=chamber,
67                             image=photo_url,
68                             name=name,
69                             party=party,
70                             district=district
71                             )
72         legislator.extras['occupation'] = occupation
73         if office_phone.strip() != "":
74             legislator.add_contact_detail(type='voice', value=office_phone, note='Capitol Office')
75 
76         # SD removed email from the detail pages but it's still in the
77         # contact page, shared for all congress people
78         member_id = re.search(r'Member=(\d+)', url).group(1)
79 
80         # find the profile block by finding a link inside it to their
81         # detail page
82         profile_link = contact_page.xpath(
83             '//ul[@id="contact-list"]//a[contains(@href, "Member=%s")]' % (member_id,))
84         if profile_link:
85             # look for the adjacent email mailto link
86             profile_link = profile_link[0]
87             profile_block = profile_link.getparent().getparent().getparent()
88             email_link = profile_block.xpath('./span/span/a[@class="mail-break"]')
89             if email_link:
90                 email = email_link[0].text
91                 email = email.lstrip()
92                 email = email.rstrip()
93                 if email:
94                     legislator.add_contact_detail(type='email',
95                                                   value=email,
96                                                   note='Capitol Office')
97         home_address = [
98                 x.strip() for x in
99                 page.xpath('//td/span[contains(@id, "HomeAddress")]/text()')
100                 if x.strip()
101                 ]
102         if home_address:
103             home_address = "\n".join(home_address)
104             home_phone = page.xpath(
105                 "string(//span[contains(@id, 'HomePhone')])").strip()
106             legislator.add_contact_detail(type='address',
107                                           value=home_address,
108                                           note='District Office')
109             if home_phone:
110                 legislator.add_contact_detail(type='voice',
111                                               value=home_phone,
112                                               note='District Office')
113 
114         legislator.add_source(url)
115         legislator.add_link(url)
116 
117         committees = page.xpath('//div[@id="divCommittees"]/span/section/table/tbody/tr/td/a')
118         for committee in committees:
119             self.scrape_committee(legislator, url, committee, chamber)
120         yield legislator
121 
122     def scrape_committee(self, leg, url, element, chamber):
123         comm = element.text.strip()
124         if comm.startswith('Joint '):
125             chamber = 'legislature'
126 
127         role = element.xpath('../following-sibling::td')[0].text_content().lower().strip()
128 
129         org = self.get_organization(comm, chamber)
130         org.add_source(url)
131         leg.add_membership(org, role=role)
132 
133     def get_organization(self, name, chamber):
134         key = (name, chamber)
135         if key not in self._committees:
136             self._committees[key] = Organization(name=name,
137                                                  chamber=chamber,
138                                                  classification='committee')
139         return self._committees[key]
140
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/openstates/sd/people.py b/openstates/sd/people.py
--- a/openstates/sd/people.py
+++ b/openstates/sd/people.py
@@ -7,17 +7,24 @@
 
 class SDLegislatorScraper(Scraper):
 
-    def scrape(self, chambers=None):
+    def scrape(self, session=None, chambers=None):
         self._committees = {}
 
+        if not session:
+            session = self.latest_session()
+            self.info('no session specified, using %s', session)
+
         # emails are on the contact page, fetch once and the
         # legislator scrapers can find their emails there
         contact_page_url = \
-            'http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx'
+            'https://sdlegislature.gov/Legislators/ContactLegislator.aspx?Session={}'.format(
+                session)
         contact_page = self.get(contact_page_url).text
         contact_page = lxml.html.fromstring(contact_page)
 
-        url = 'http://www.sdlegislature.gov/Legislators/default.aspx'
+        # https://sdlegislature.gov/Legislators/default.aspx?Session=2018
+        url = 'https://sdlegislature.gov/Legislators/default.aspx?Session={}'.format(
+            session)
         if chambers is None:
             chambers = ['upper', 'lower']
         for chamber in chambers:
@@ -71,7 +78,8 @@
                             )
         legislator.extras['occupation'] = occupation
         if office_phone.strip() != "":
-            legislator.add_contact_detail(type='voice', value=office_phone, note='Capitol Office')
+            legislator.add_contact_detail(
+                type='voice', value=office_phone, note='Capitol Office')
 
         # SD removed email from the detail pages but it's still in the
         # contact page, shared for all congress people
@@ -85,7 +93,8 @@
             # look for the adjacent email mailto link
             profile_link = profile_link[0]
             profile_block = profile_link.getparent().getparent().getparent()
-            email_link = profile_block.xpath('./span/span/a[@class="mail-break"]')
+            email_link = profile_block.xpath(
+                './span/span/a[@class="mail-break"]')
             if email_link:
                 email = email_link[0].text
                 email = email.lstrip()
@@ -95,10 +104,10 @@
                                                   value=email,
                                                   note='Capitol Office')
         home_address = [
-                x.strip() for x in
-                page.xpath('//td/span[contains(@id, "HomeAddress")]/text()')
-                if x.strip()
-                ]
+            x.strip() for x in
+            page.xpath('//td/span[contains(@id, "HomeAddress")]/text()')
+            if x.strip()
+        ]
         if home_address:
             home_address = "\n".join(home_address)
             home_phone = page.xpath(
@@ -114,7 +123,8 @@
         legislator.add_source(url)
         legislator.add_link(url)
 
-        committees = page.xpath('//div[@id="divCommittees"]/span/section/table/tbody/tr/td/a')
+        committees = page.xpath(
+            '//div[@id="divCommittees"]/span/section/table/tbody/tr/td/a')
         for committee in committees:
             self.scrape_committee(legislator, url, committee, chamber)
         yield legislator
@@ -124,7 +134,8 @@
         if comm.startswith('Joint '):
             chamber = 'legislature'
 
-        role = element.xpath('../following-sibling::td')[0].text_content().lower().strip()
+        role = element.xpath(
+            '../following-sibling::td')[0].text_content().lower().strip()
 
         org = self.get_organization(comm, chamber)
         org.add_source(url)

{"golden_diff": "diff --git a/openstates/sd/people.py b/openstates/sd/people.py\n--- a/openstates/sd/people.py\n+++ b/openstates/sd/people.py\n@@ -7,17 +7,24 @@\n \n class SDLegislatorScraper(Scraper):\n \n-    def scrape(self, chambers=None):\n+    def scrape(self, session=None, chambers=None):\n         self._committees = {}\n \n+        if not session:\n+            session = self.latest_session()\n+            self.info('no session specified, using %s', session)\n+\n         # emails are on the contact page, fetch once and the\n         # legislator scrapers can find their emails there\n         contact_page_url = \\\n-            'http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx'\n+            'https://sdlegislature.gov/Legislators/ContactLegislator.aspx?Session={}'.format(\n+                session)\n         contact_page = self.get(contact_page_url).text\n         contact_page = lxml.html.fromstring(contact_page)\n \n-        url = 'http://www.sdlegislature.gov/Legislators/default.aspx'\n+        # https://sdlegislature.gov/Legislators/default.aspx?Session=2018\n+        url = 'https://sdlegislature.gov/Legislators/default.aspx?Session={}'.format(\n+            session)\n         if chambers is None:\n             chambers = ['upper', 'lower']\n         for chamber in chambers:\n@@ -71,7 +78,8 @@\n                             )\n         legislator.extras['occupation'] = occupation\n         if office_phone.strip() != \"\":\n-            legislator.add_contact_detail(type='voice', value=office_phone, note='Capitol Office')\n+            legislator.add_contact_detail(\n+                type='voice', value=office_phone, note='Capitol Office')\n \n         # SD removed email from the detail pages but it's still in the\n         # contact page, shared for all congress people\n@@ -85,7 +93,8 @@\n             # look for the adjacent email mailto link\n             profile_link = profile_link[0]\n             profile_block = profile_link.getparent().getparent().getparent()\n-            email_link = profile_block.xpath('./span/span/a[@class=\"mail-break\"]')\n+            email_link = profile_block.xpath(\n+                './span/span/a[@class=\"mail-break\"]')\n             if email_link:\n                 email = email_link[0].text\n                 email = email.lstrip()\n@@ -95,10 +104,10 @@\n                                                   value=email,\n                                                   note='Capitol Office')\n         home_address = [\n-                x.strip() for x in\n-                page.xpath('//td/span[contains(@id, \"HomeAddress\")]/text()')\n-                if x.strip()\n-                ]\n+            x.strip() for x in\n+            page.xpath('//td/span[contains(@id, \"HomeAddress\")]/text()')\n+            if x.strip()\n+        ]\n         if home_address:\n             home_address = \"\\n\".join(home_address)\n             home_phone = page.xpath(\n@@ -114,7 +123,8 @@\n         legislator.add_source(url)\n         legislator.add_link(url)\n \n-        committees = page.xpath('//div[@id=\"divCommittees\"]/span/section/table/tbody/tr/td/a')\n+        committees = page.xpath(\n+            '//div[@id=\"divCommittees\"]/span/section/table/tbody/tr/td/a')\n         for committee in committees:\n             self.scrape_committee(legislator, url, committee, chamber)\n         yield legislator\n@@ -124,7 +134,8 @@\n         if comm.startswith('Joint '):\n             chamber = 'legislature'\n \n-        role = element.xpath('../following-sibling::td')[0].text_content().lower().strip()\n+        role = element.xpath(\n+            '../following-sibling::td')[0].text_content().lower().strip()\n \n         org = self.get_organization(comm, chamber)\n         org.add_source(url)\n", "issue": "SD failing since at least 2018-07-25\nSD has been failing since 2018-07-25\n\nBased on automated runs it appears that SD has not run successfully in 2 days (2018-07-25).\n\n\n```\n  02:01:45 INFO pupa: save post 9 as post_c1c5c0f0-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 10 as post_c1c5c244-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 11 as post_c1c5c38e-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 12 as post_c1c5c4d8-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 13 as post_c1c5c622-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 14 as post_c1c5c76c-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 15 as post_c1c5c8ac-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 16 as post_c1c5c9ec-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 17 as post_c1c5cb36-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 18 as post_c1c5cc76-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 19 as post_c1c5cdc0-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 20 as post_c1c5cf00-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 21 as post_c1c5d04a-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 22 as post_c1c5d18a-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 23 as post_c1c5d2d4-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 24 as post_c1c5d41e-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 25 as post_c1c5d55e-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 27 as post_c1c5d69e-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 29 as post_c1c5d7e8-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 30 as post_c1c5d9f0-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 31 as post_c1c5dbb2-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 32 as post_c1c5dcf2-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 33 as post_c1c5de5a-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 34 as post_c1c5dfc2-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 35 as post_c1c5e13e-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 26A as post_c1c5e292-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 26B as post_c1c5e3d2-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 28A as post_c1c5e508-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO pupa: save post 28B as post_c1c5e648-90a1-11e8-8ead-0244d764d5a8.json\n02:01:45 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx\n02:01:45 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/default.aspx\n02:01:46 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/default.aspx\nloaded Open States pupa settings...\nsd (scrape, import)\n  people: {}\n  bills: {}\nTraceback (most recent call last):\n  File \"/opt/openstates/venv-pupa//bin/pupa\", line 11, in <module>\n    load_entry_point('pupa', 'console_scripts', 'pupa')()\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py\", line 68, in main\n    subcommands[args.subcommand].handle(args, other)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py\", line 260, in handle\n    return self.do_handle(args, other, juris)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py\", line 305, in do_handle\n    report['scrape'] = self.do_scrape(juris, args, scrapers)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py\", line 173, in do_scrape\n    report[scraper_name] = scraper.do_scrape(**scrape_args)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py\", line 120, in do_scrape\n    raise ScrapeError('no objects returned from {} scrape'.format(self.__class__.__name__))\npupa.exceptions.ScrapeError: no objects returned from SDLegislatorScraper scrape\n```\n\nVisit http://bobsled.openstates.org for more info.\n\n", "before_files": [{"content": "import re\n\nfrom pupa.scrape import Person, Scraper, Organization\n\nimport lxml.html\n\n\nclass SDLegislatorScraper(Scraper):\n\n    def scrape(self, chambers=None):\n        self._committees = {}\n\n        # emails are on the contact page, fetch once and the\n        # legislator scrapers can find their emails there\n        contact_page_url = \\\n            'http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx'\n        contact_page = self.get(contact_page_url).text\n        contact_page = lxml.html.fromstring(contact_page)\n\n        url = 'http://www.sdlegislature.gov/Legislators/default.aspx'\n        if chambers is None:\n            chambers = ['upper', 'lower']\n        for chamber in chambers:\n            if chamber == 'upper':\n                search = 'Senate Legislators'\n            else:\n                search = 'House Legislators'\n\n            page = self.get(url).text\n            page = lxml.html.fromstring(page)\n            page.make_links_absolute(url)\n\n            # Legisltor listing has initially-hidden <div>s that\n            # contain the members for just a particular chamber\n            for link in page.xpath(\n                \"//h4[text()='{}']\".format(search) +\n                \"/../span/section/table/tbody/tr/td/a\"\n            ):\n                name = link.text.strip()\n                yield from self.scrape_legislator(name, chamber, link.attrib['href'], contact_page)\n        yield from self._committees.values()\n\n    def scrape_legislator(self, name, chamber, url, contact_page):\n        page = self.get(url).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(url)\n\n        party = page.xpath(\"string(//span[contains(@id, 'Party')])\")\n        party = party.strip()\n\n        if party == 'Democrat':\n            party = 'Democratic'\n\n        district = page.xpath(\"string(//span[contains(@id, 'District')])\")\n        district = district.strip().lstrip('0')\n\n        occupation = page.xpath(\n            \"string(//span[contains(@id, 'Occupation')])\")\n        occupation = occupation.strip()\n\n        (photo_url, ) = page.xpath('//img[contains(@id, \"_imgMember\")]/@src')\n\n        office_phone = page.xpath(\n            \"string(//span[contains(@id, 'CapitolPhone')])\").strip()\n\n        legislator = Person(primary_org=chamber,\n                            image=photo_url,\n                            name=name,\n                            party=party,\n                            district=district\n                            )\n        legislator.extras['occupation'] = occupation\n        if office_phone.strip() != \"\":\n            legislator.add_contact_detail(type='voice', value=office_phone, note='Capitol Office')\n\n        # SD removed email from the detail pages but it's still in the\n        # contact page, shared for all congress people\n        member_id = re.search(r'Member=(\\d+)', url).group(1)\n\n        # find the profile block by finding a link inside it to their\n        # detail page\n        profile_link = contact_page.xpath(\n            '//ul[@id=\"contact-list\"]//a[contains(@href, \"Member=%s\")]' % (member_id,))\n        if profile_link:\n            # look for the adjacent email mailto link\n            profile_link = profile_link[0]\n            profile_block = profile_link.getparent().getparent().getparent()\n            email_link = profile_block.xpath('./span/span/a[@class=\"mail-break\"]')\n            if email_link:\n                email = email_link[0].text\n                email = email.lstrip()\n                email = email.rstrip()\n                if email:\n                    legislator.add_contact_detail(type='email',\n                                                  value=email,\n                                                  note='Capitol Office')\n        home_address = [\n                x.strip() for x in\n                page.xpath('//td/span[contains(@id, \"HomeAddress\")]/text()')\n                if x.strip()\n                ]\n        if home_address:\n            home_address = \"\\n\".join(home_address)\n            home_phone = page.xpath(\n                \"string(//span[contains(@id, 'HomePhone')])\").strip()\n            legislator.add_contact_detail(type='address',\n                                          value=home_address,\n                                          note='District Office')\n            if home_phone:\n                legislator.add_contact_detail(type='voice',\n                                              value=home_phone,\n                                              note='District Office')\n\n        legislator.add_source(url)\n        legislator.add_link(url)\n\n        committees = page.xpath('//div[@id=\"divCommittees\"]/span/section/table/tbody/tr/td/a')\n        for committee in committees:\n            self.scrape_committee(legislator, url, committee, chamber)\n        yield legislator\n\n    def scrape_committee(self, leg, url, element, chamber):\n        comm = element.text.strip()\n        if comm.startswith('Joint '):\n            chamber = 'legislature'\n\n        role = element.xpath('../following-sibling::td')[0].text_content().lower().strip()\n\n        org = self.get_organization(comm, chamber)\n        org.add_source(url)\n        leg.add_membership(org, role=role)\n\n    def get_organization(self, name, chamber):\n        key = (name, chamber)\n        if key not in self._committees:\n            self._committees[key] = Organization(name=name,\n                                                 chamber=chamber,\n                                                 classification='committee')\n        return self._committees[key]\n", "path": "openstates/sd/people.py"}], "after_files": [{"content": "import re\n\nfrom pupa.scrape import Person, Scraper, Organization\n\nimport lxml.html\n\n\nclass SDLegislatorScraper(Scraper):\n\n    def scrape(self, session=None, chambers=None):\n        self._committees = {}\n\n        if not session:\n            session = self.latest_session()\n            self.info('no session specified, using %s', session)\n\n        # emails are on the contact page, fetch once and the\n        # legislator scrapers can find their emails there\n        contact_page_url = \\\n            'https://sdlegislature.gov/Legislators/ContactLegislator.aspx?Session={}'.format(\n                session)\n        contact_page = self.get(contact_page_url).text\n        contact_page = lxml.html.fromstring(contact_page)\n\n        # https://sdlegislature.gov/Legislators/default.aspx?Session=2018\n        url = 'https://sdlegislature.gov/Legislators/default.aspx?Session={}'.format(\n            session)\n        if chambers is None:\n            chambers = ['upper', 'lower']\n        for chamber in chambers:\n            if chamber == 'upper':\n                search = 'Senate Legislators'\n            else:\n                search = 'House Legislators'\n\n            page = self.get(url).text\n            page = lxml.html.fromstring(page)\n            page.make_links_absolute(url)\n\n            # Legisltor listing has initially-hidden <div>s that\n            # contain the members for just a particular chamber\n            for link in page.xpath(\n                \"//h4[text()='{}']\".format(search) +\n                \"/../span/section/table/tbody/tr/td/a\"\n            ):\n                name = link.text.strip()\n                yield from self.scrape_legislator(name, chamber, link.attrib['href'], contact_page)\n        yield from self._committees.values()\n\n    def scrape_legislator(self, name, chamber, url, contact_page):\n        page = self.get(url).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(url)\n\n        party = page.xpath(\"string(//span[contains(@id, 'Party')])\")\n        party = party.strip()\n\n        if party == 'Democrat':\n            party = 'Democratic'\n\n        district = page.xpath(\"string(//span[contains(@id, 'District')])\")\n        district = district.strip().lstrip('0')\n\n        occupation = page.xpath(\n            \"string(//span[contains(@id, 'Occupation')])\")\n        occupation = occupation.strip()\n\n        (photo_url, ) = page.xpath('//img[contains(@id, \"_imgMember\")]/@src')\n\n        office_phone = page.xpath(\n            \"string(//span[contains(@id, 'CapitolPhone')])\").strip()\n\n        legislator = Person(primary_org=chamber,\n                            image=photo_url,\n                            name=name,\n                            party=party,\n                            district=district\n                            )\n        legislator.extras['occupation'] = occupation\n        if office_phone.strip() != \"\":\n            legislator.add_contact_detail(\n                type='voice', value=office_phone, note='Capitol Office')\n\n        # SD removed email from the detail pages but it's still in the\n        # contact page, shared for all congress people\n        member_id = re.search(r'Member=(\\d+)', url).group(1)\n\n        # find the profile block by finding a link inside it to their\n        # detail page\n        profile_link = contact_page.xpath(\n            '//ul[@id=\"contact-list\"]//a[contains(@href, \"Member=%s\")]' % (member_id,))\n        if profile_link:\n            # look for the adjacent email mailto link\n            profile_link = profile_link[0]\n            profile_block = profile_link.getparent().getparent().getparent()\n            email_link = profile_block.xpath(\n                './span/span/a[@class=\"mail-break\"]')\n            if email_link:\n                email = email_link[0].text\n                email = email.lstrip()\n                email = email.rstrip()\n                if email:\n                    legislator.add_contact_detail(type='email',\n                                                  value=email,\n                                                  note='Capitol Office')\n        home_address = [\n            x.strip() for x in\n            page.xpath('//td/span[contains(@id, \"HomeAddress\")]/text()')\n            if x.strip()\n        ]\n        if home_address:\n            home_address = \"\\n\".join(home_address)\n            home_phone = page.xpath(\n                \"string(//span[contains(@id, 'HomePhone')])\").strip()\n            legislator.add_contact_detail(type='address',\n                                          value=home_address,\n                                          note='District Office')\n            if home_phone:\n                legislator.add_contact_detail(type='voice',\n                                              value=home_phone,\n                                              note='District Office')\n\n        legislator.add_source(url)\n        legislator.add_link(url)\n\n        committees = page.xpath(\n            '//div[@id=\"divCommittees\"]/span/section/table/tbody/tr/td/a')\n        for committee in committees:\n            self.scrape_committee(legislator, url, committee, chamber)\n        yield legislator\n\n    def scrape_committee(self, leg, url, element, chamber):\n        comm = element.text.strip()\n        if comm.startswith('Joint '):\n            chamber = 'legislature'\n\n        role = element.xpath(\n            '../following-sibling::td')[0].text_content().lower().strip()\n\n        org = self.get_organization(comm, chamber)\n        org.add_source(url)\n        leg.add_membership(org, role=role)\n\n    def get_organization(self, name, chamber):\n        key = (name, chamber)\n        if key not in self._committees:\n            self._committees[key] = Organization(name=name,\n                                                 chamber=chamber,\n                                                 classification='committee')\n        return self._committees[key]\n", "path": "openstates/sd/people.py"}]}