title: IndexError: list index out of range, When I run two python scripts at the same time
body: <p>I want to crawl data from a website, which is sorted by different province. To be faster, I try to run two or more python scripts to crawl each province at the same time. The only difference between two scripts is they crawl different url sets. Every time they worked well at the first 30 seconds or 1 minute. But later I got the below error in each script, and they arised simultaneously every time:</p>
  
  <pre><code>Traceback (most recent call last):
    File "EOLGrades-A.py", line 157, in &lt;module&gt;
      dealEachCollege(url_2,tableName2,cNameList[i])
    File "EOLGrades-A.py", line 58, in dealEachCollege
      insertData(getData(sp),tableName,collegeName)
    File "EOLGrades-A.py", line 33, in getData
      if FAtd[x+j].text == '--' or FAtd[x+j].text ==' ':
  IndexError: list index out of range
  
  Traceback (most recent call last):
    File "EOLGrades-B.py", line 157, in &lt;module&gt;
      dealEachCollege(url_2,tableName2,cNameList[i])
    File "EOLGrades-B.py", line 58, in dealEachCollege
      insertData(getData(sp),tableName,collegeName)
    File "EOLGrades-B.py", line 33, in getData
      if FAtd[x+j].text == '--' or FAtd[x+j].text ==' ':
  IndexError: list index out of range
  </code></pre>
  
  <p>My getData Method:</p>
  
  <pre><code>def getData(soup,count):
      FAtr = soup.find_all(name='tr')
      FAtd = soup.find_all(name='td')
      m = [([0] * 6) for i in range(count)]
      x = 0
      if(len(FAtd)&lt;=1):
          print("no data")
          return ['0']
      else:      
          for i in range(count):
              for j in range(6):
                  if FAtd[x+j].text == '--' or FAtd[x+j].text ==' ':
                      m[i][j] = None
                  else:
                      content = FAtd[x+j].text.strip()
                      m[i][j] = content           
              x= x+6
      return m
  </code></pre>
  
  <p>count is from getCount Method:</p>
  
  <pre><code>def getCount(soup):
      FAtr = soup.find_all(name='tr')
      count = len(FAtr) - 1
      return count
  </code></pre>
  
  <p>dealEachCollege Method:</p>
  
  <pre><code>def dealEachCollege(URL,tableName,collegeName):
      page = s.get(URL,headers=headers)
      page.encoding='utf-8'
      sp = BeautifulSoup(page.text,"html.parser")
      count = getCount(sp)
      insertData(getData(sp,count),tableName,collegeName,count)
      page.close()
  </code></pre>
  
  <p>insertData Method:</p>
  
  <pre><code>def insertData(dataList,tableName,collegeName,count):
      try:
          m=dataList
          if m[0]=='0':
              return
          for i in range(count):
              cursor.execute("INSERT INTO " + tableName + " VALUES (%s,%s,%s,%s,%s,%s,%s)",(collegeName,m[i][0],m[i][1],m[i][2],m[i][3],m[i][4],m[i][5]))
              conn.commit() 
          print ("Successfully inserted into %s." %tableName)
      except pymysql.Error as e:
          print ("Mysql Error %d: %s" %(e.args[0], e.args[1]))
  </code></pre>
  
  <p>The error didn't arise when I just ran one single script.
  Could anybody tell me how to fix it? Or any other methods to run 2 python crawlers at the same time? Many thanks!</p>