title: Reading URLs from .csv and appending scrape results below previous with Python, BeautifulSoup, Pandas
body: <p>I got this code to almost work, despite much ignorance. Please help on the home run!</p>
  <ul>
  <li>Problem 1: INPUT:</li>
  </ul>
  <p>I have a long list of URLs (1000+) to read from and they are in a single column in .csv. I would prefer to read from that file than to paste them into code, like below.</p>
  <ul>
  <li>Problem 2: OUTPUT:</li>
  </ul>
  <p>The source files actually have 3 drivers and 3 challenges each. In a separate python file, the below code finds, prints and saves all 3, but not when I'm using this dataframe below (see below - it only saves 2).</p>
  <ul>
  <li>Problem 3: OUTPUT:</li>
  </ul>
  <p>I want the output (both files) to have URLs in column 0, and then drivers (or challenges) in the following columns. But what I've written here (probably the 'drop') makes them not only drop one row but also move across 2 columns.</p>
  <p>At the end I'm showing both the inputs and the current &amp; desired output. Sorry for the long question. I'll be very grateful for any help!</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  import pandas as pd
  
  urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
  dataframes = []
  dataframes2 = []
  
  for url in urls:
      page = requests.get(url)
      soup = BeautifulSoup(page.text, 'html.parser')
      toc = soup.find(&quot;div&quot;, id=&quot;toc&quot;)
  
      def get_drivers():
          data = []
          for x in toc.select('li:-soup-contains-own(&quot;Market drivers&quot;) li'):
              data.append(x.get_text(strip=True))
          df = pd.DataFrame(data, columns=[url])
          dataframes.append(pd.DataFrame(df).drop(0, axis=0))
          df2 = pd.concat(dataframes)
          tdata = df2.T
          tdata.to_csv(f'detail-dr.csv', header=True)
  
      get_drivers()
  
  
      def get_challenges():
          data = []
          for y in toc.select('li:-soup-contains-own(&quot;Market challenges&quot;) li'):
              data.append(y.get_text(strip=True).replace('Table Impact of drivers and challenges', ''))
          df = pd.DataFrame(data, columns=[url])
          dataframes2.append(pd.DataFrame(df).drop(0, axis=0))
          df2 = pd.concat(dataframes2)
          tdata = df2.T
          tdata.to_csv(f'detail-ch.csv', header=True)
  
      get_challenges()
  </code></pre>
  <p><strong>The inputs look like this in each URL. They are just lists:</strong></p>
  <p>Market drivers</p>
  <ul>
  <li>Growing investment in fabs</li>
  <li>Miniaturization of electronic products</li>
  <li>Increasing demand for IoT devices</li>
  </ul>
  <p>Market challenges</p>
  <ul>
  <li>Rapid technological changes in semiconductor industry</li>
  <li>Volatility in semiconductor industry</li>
  <li>Impact of technology chasm Table Impact of drivers and challenges</li>
  </ul>
  <p><strong>My desired output for drivers is:</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">0</th>
  <th style="text-align: left;">1</th>
  <th style="text-align: left;">2</th>
  <th style="text-align: left;">3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">http/.../Global-Induction-Hobs-30196623/</td>
  <td style="text-align: left;">Product innovations and new designs</td>
  <td style="text-align: left;">Increasing demand for convenient home appliances with changes in lifestyle patterns</td>
  <td style="text-align: left;">Growing adoption of energy-efficient appliances</td>
  </tr>
  <tr>
  <td style="text-align: left;">http/.../Global-Human-Capital-Management-30196628/</td>
  <td style="text-align: left;">Demand for automated recruitment processes</td>
  <td style="text-align: left;">Increasing demand for unified solutions for all HR functions</td>
  <td style="text-align: left;">Increasing workforce diversity</td>
  </tr>
  <tr>
  <td style="text-align: left;">http/.../Global-Probe-Card-30196643/</td>
  <td style="text-align: left;">Growing investment in fabs</td>
  <td style="text-align: left;">Miniaturization of electronic products</td>
  <td style="text-align: left;">Increasing demand for IoT devices</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>But instead I get:</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">0</th>
  <th style="text-align: left;">1</th>
  <th style="text-align: left;">2</th>
  <th style="text-align: left;">3</th>
  <th style="text-align: left;">4</th>
  <th style="text-align: left;">5</th>
  <th style="text-align: left;">6</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">http/.../Global-Induction-Hobs-30196623/</td>
  <td style="text-align: left;">Increasing demand for convenient home appliances with changes in lifestyle patterns</td>
  <td style="text-align: left;">Growing adoption of energy-efficient appliances</td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  </tr>
  <tr>
  <td style="text-align: left;">http/.../Global-Human-Capital-Management-30196628/</td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;">Increasing demand for unified solutions for all HR functions</td>
  <td style="text-align: left;">Increasing workforce diversity</td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  </tr>
  <tr>
  <td style="text-align: left;">http/.../Global-Probe-Card-30196643/</td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;"></td>
  <td style="text-align: left;">Miniaturization of electronic products</td>
  <td style="text-align: left;">Increasing demand for IoT devices</td>
  </tr>
  </tbody>
  </table>
  </div>