title: How to web scrape a list of URLs of a website with multiprocessing when I login using Python
body: <p>First of all I am a beginner with Python. Now I am trying to create a script that does the following</p>
  <ol>
  <li>login to a website using Selenium</li>
  <li>load a list of the website's URLs from a CSV file</li>
  <li>web scrape data using multiprocessing method</li>
  </ol>
  <p>I am using the following script</p>
  <pre><code>#Load URLS from CSV
  def mycontents():
      contents = []
      with open('global_csv.csv', 'r') as csvf:
          reader = csv.reader(csvf, delimiter=&quot;;&quot;)
          for row in reader:
              contents.append(row[1]) # Add each url to list contents
      return contents
  # parse a single item to get information
  def parse(url):   
  
  headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'}  
  
  r = requests.get(url, headers, timeout=10)
  sleep(3)
  
  info = []
  availability_text = '-'
  price_text = '-'
  
  
  if r.status_code == 200:
      print('Processing..'+ url)
      html = r.text
      
      soup = BeautifulSoup(html, 'html.parser')
      time.sleep(4)
      price = soup.select(&quot;.price&quot;)
            
      if price is not None:
       price_text = price.text.strip()
       print(price_text)
      else:
       price_text = &quot;0,00&quot;
       print(price_text)
  
      availability = soup.find('span', attrs={'class':'wholesale-availability'})
      
      if availability is not None:
       availability_text = availability.text.strip()
       print(availability_text)
      else:
       availability_text = &quot;Not Available&quot;
       print(availability_text)
  
      info.append(price_text)
      info.append(availability_text)        
  
  return ';'.join(info)
  
  web_links = None
  web_links = mycontents()
  
  #Insert First Row
  fields=['SKU','price','availability']
  with open('output_global.csv', 'w') as f:
      writer = csv.writer(f)
      writer.writerow(fields)
  
  if __name__ == &quot;__main__&quot;:
      #Load Webdriver
      browser = webdriver.Chrome('C:\\chromedriver.exe')
      browser.get('TheLoginPage')
      #Find Username Field
      username = browser.find_element_by_id('email')
      username.send_keys('myusername')
      #Find Password Field
      password = browser.find_element_by_id('pass')
      time.sleep(2)
      password.send_keys('mypassword')
      #Find Connect Button
      sing_in = browser.find_element_by_xpath('//*[@id=&quot;send2&quot;]')
      sing_in.click()
      #Start MultiProcess
      with Pool(4) as p:    
        records = p.map(parse, web_links)
  
      if len(records) &gt; 0:
           with open('output_global.csv', 'a') as f:
            f.write('\n'.join(records))
  </code></pre>
  <p>When I run the script is not getting anything and in Command Window it is just shows the URLs, which makes me think that even if I connect successfully the sessions are different?!</p>
  <p>I tried to save the session by putting it inside parse method or</p>
  <pre><code>if __name__ == &quot;__main__&quot;:
  </code></pre>
  <p>I tried to connect to the browser the same session but I get errors like</p>
  <ol>
  <li>You have not defined a session</li>
  <li>TypeError: get() takes 2 positional arguments but 3 were given</li>
  <li>local variable 'session' referenced before assignment</li>
  </ol>
  <p>How can I practically login to the website and <strong>use multiprocessing</strong> to web scrape the URLs I need?</p>