title: BeautifulSoup: IndexError: list index out of range for Multiple Links to Scrape
body: <p>I am working on a scraper, it works well on multiple URLs (it takes some basic info and multiple links associated with each URL), BUT since each URL has a different number of links associated with each profile (e.g. one URL can have 3 social links, another URL only 1 social link, then another URL with 2 social links) whenever I run it, as soon as the scraper encounters a URL without enough social links, it says &quot;social2 = links[1] IndexError: list index out of range&quot;</p>
  <p>I am pretty sure I have to create a &quot;IF&quot; that tells the scraper to still scrape all the other info but to skip a column if the data (in this case, social links) is not present or are less then 2,3,4, etc so that if a URL has only 1 link or 2 links but not all the one I am calling, the scraper should write &quot;None&quot; for that specific link</p>
  <p>Also, I'd like to tell the scraper to continue looking for all social links until the URL doesn't have anymore because I don't know how many MAX links each URL can actually have.</p>
  <p>I'd like to see something like this in my CSV:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Name</th>
  <th>Location</th>
  <th>Link</th>
  <th>Link 2</th>
  <th>Link 3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Mark Red</td>
  <td>Los Angeles</td>
  <td><a href="https://instagram.com/markred" rel="nofollow noreferrer">https://instagram.com/markred</a></td>
  <td>None</td>
  <td><a href="https://tiktok.com/@markred" rel="nofollow noreferrer">https://tiktok.com/@markred</a></td>
  </tr>
  <tr>
  <td>Mary Green</td>
  <td>New York</td>
  <td><a href="https://instagram.com/marygreen" rel="nofollow noreferrer">https://instagram.com/marygreen</a></td>
  <td><a href="https://youtuebe.com/marygreen" rel="nofollow noreferrer">https://youtuebe.com/marygreen</a></td>
  <td>None</td>
  </tr>
  <tr>
  <td></td>
  <td></td>
  <td></td>
  <td></td>
  <td></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>My code is:</p>
  <pre><code>from bs4 import BeautifulSoup
  import requests
  from csv import writer
  
  urls = ['https://url.com/1','https://url.com/2', 'https://url.com/3']
  
  with open('multi.csv', 'w', encoding='utf8', newline='') as f:
      thewriter = writer(f)
      header = ['Name', 'Location', 'Link', 'Link2', 'Link3']
      thewriter.writerow(header)
  
      for url in urls:
          my_url = requests.get(url)
          html = my_url.content
          soup = BeautifulSoup(html,'html.parser')
  
          info = []
  
          lists = soup.find_all('div', class_=&quot;profile-info-holder&quot;)
  
          for l in lists:
              name = l.find('div', class_=&quot;profile-name&quot;).text
              location = l.find('div', class_=&quot;profile-location&quot;).text
              links = l.find_all('a', class_=&quot;intercept&quot;, href=True)
              social1 = links[0]
              social2 = links[1]
              social3 = links[2]
          
              info = [name, location, social1.get('href'), social2.get('href'), social3.get('href')]
              thewriter.writerow(info)
  </code></pre>
  <p>Thank you!</p>