title: to add extraction code to beautifulsoup loop
body: <p>I have a code that extracts 2 links from txt file for example</p>
  
  <pre><code>https://www.kununu.com/de/volkswagenconsulting/kommentare
  https://www.kununu.com/de/volkswagennutzfahrzeuge/kommentare
  </code></pre>
  
  <p>Below code extracts from that links arbeitsatmosphare data and firma data (code works).</p>
  
  <pre><code>firma=[]
  
  company=[]
  
  arbeitsatmosphare = []
  
  lineList2=[]
  
   with open((os.environ.get('mypath')+'/test.txt'), 'r') as file:
  
      lines = file.readlines()
      for line in lines:
          lineList2.append(line.strip('\n'))
  
  print(lineList2)
  
  
      for lurl in lineList2:
      with requests.Session() as session:
          session.headers = {
              'x-requested-with': 'XMLHttpRequest'
          }
          page = 1
          while True:
              print(f"Processing page {page}..")
              url = f'{lurl}/{page}'
              print(url)
              response = session.get(url)
              soup = BeautifulSoup(response.text, 'html.parser')
              articles = soup.find_all('article')
              print("Number of articles: " + str(len(articles)))
              for article in articles:
  
                  try:
                      arbeitsatmosphareText = article.find('span', text=re.compile(r'Arbeitsatmosphäre')).find_next(
                          'span').text.strip()
                      arbeitsatmosphare.append(arbeitsatmosphareText)
                  except:
                      arbeitsatmosphare.append('N/A')
  
                  try:
                      firmaText = article.find('div', text=re.compile(r'Firma')).find_next('div').text.strip()
                      firma.append(firmaText)
                  except:
                      firma.append('N/A')
                  companyText = [x.text.strip() for x in soup.find_all('span', {'class': 'company-name'})]
                  company.append(companyText)
  
              page += 1
              pagination = soup.find_all('div', {'class': 'paginationControl'})
              if not pagination:
                  break
  
      df = pd.DataFrame({'Arbeitsatmosphäre': arbeitsatmosphare,
                         'firma':firma
                         'company': company
  
                         })
  
      print(df)
  </code></pre>
  
  <p>I need to include company information into output dataframe. For example "Volkswagen Nutzfahrzeuge". Company data is located in the page header below black stripe.</p>
  
  <p>UPD I updated code, it works now.</p>