title: Fetching lawyer details from different links of a website using bs4 in python
body: <p>I am an absolute beginner to Web Scraping using Python with very little knowledge in Python programming. I am just trying to extract the information of the lawyers in the Tennesse location. In the webpage ,there are multiple links, within which there are further more links about the categories of lawyers and within those are the lawyers details.</p>
  
  <p>I have already extracted the links of the various city into a list and have also extracted the various categories of lawyers available in each of the city links. Furthermore I have extracted each of the lawyers url in separate categories and stored them in a separate set. Now I have iterated over those urls for fetching the details of the lawyers and finally writing them on to a .xls file. But while I am running my code the process is going on executing for and indefinite amount of time. I have to force stop the process and even no .xls file is being created at the desired location as stated in the program. What can be done? If possible suggest.</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup as bs
  import pandas as pd
  
  final=[]
  records=[]
  with requests.Session() as s:
      res = s.get('https://attorneys.superlawyers.com/tennessee/', headers = {'User-agent': 'Super Bot 9000'})
      soup = bs(res.content, 'lxml')
  
      cities = [item['href'] for item in soup.select('#browse_view a')]
      for c in cities:
          r=s.get(c)
          s1=bs(r.content,'lxml')
          categories = [item['href'] for item in s1.select('.three_browse_columns:nth-of-type(2) a')]
          for c1 in categories:
              r1=s.get(c1)
              s2=bs(r1.content,'lxml')
              lawyers = [item['href'].split('*')[1] if '*' in item['href'] else item['href'] for item in
                         s2.select('.indigo_text .directory_profile')]
  
  
  
              for i in lawyers:
                  r2 = s.get(i)
                  s3 = bs(r2.content, 'lxml')
                  name = s3.select_one('#lawyer_name').text
                  category = s3.select_one('#attorney_profile_heading').text
                  firm = s3.select_one('#firm_profile_page').text
                  address = ' '.join([string for string in s3.select_one('#poap_postal_addr_block').stripped_strings][1:])
                  practices = ' '.join([item.text for item in s3.select('#pa_list li')])
                  records.append({'Names': name,'Category':category,'Address': address, 'Firm Name': firm,'Practice Area':practices})
  df = pd.DataFrame(records,columns=['Names','Category','Address','Farm Name','Practice Areas'])
  df=df.drop_duplicates()
  df.to_excel(r'C:\Users\laptop\Desktop\lawyers.xls', sheet_name='MyData2', index = False, header=True)
  
  </code></pre>
  
  <p>"I expected the program to complete its execution and create an .xls file, but it is going on executing and even I have no idea for how long will it require to complete it's execution. Is there any possibility that an infinite loop has occoured? If possible suggest."</p>