Title: How to add a password log in into screen scraping with Beautiful Soup / Python
Tags: python
Question: I am trying to figure out how to add a piece of code that will help the code I already have to sign into the website so that my program can screen scrape the data I need. The sign in is a simple "Log in ID" and "Password". After sign in I am redirected to a page that is not the page I want to scrape data from. 

I have been unable to test my code. My code always gives me "None". I assuming this is due to it not being able to sign into the website. 

I am trying to screen scrape:

```<div class="Object7069"&gt;
  <div style="font-family:verdana; font-size:9px; text-align:right; color:#999999;"&gt;
    124 of 256
  </div&gt;
</div&gt;  
```

It is a simple variable "124/256" that will change over time. 

My code:

```import urllib2
from bs4 import BeautifulSoup
quote_page = 'https://admin252.acellus.com/StudentFunctions/progress.html'
page = urllib2.urlopen(quote_page)
soup = BeautifulSoup(page, 'html.parser')
price_box = soup.find('div', attrs={'class':'object7069'})
price = price_box
print price
```

Comment: Just realized it deleted the first half of my question...

Comment: What **exactly** is your question?


Here is another answer: Without actually logging in, I can't test the code properly but when I login to a different website and scrape data (which I do all the time) I use the requests library because it saves cookies and other login info.

Something like this:

```import requests
from bs4 import BeautifulSoup

quote_page = 'https://admin252.acellus.com/StudentFunctions/progress.html'
login_params = {'LoginUsername': 'Kyle', 'Password': 'abcd'}
# You will need to find out which params are actually sent and if it's a GET or POST request. 
# I use Firefox developer (F12) to snoop on the parameters that are sent while logging in.
resp = requests.get(quote_page, params=login_params)
soup = BeautifulSoup(resp.content, 'html.parser')
price_box = soup.find('div', attrs={'class':'object7069'})
price = price_box.get_text()
print price
```

BTW, I find myself testing screen scraping in Python's interactive mode before running it as a script. For example, I'd print the resp in order to see if it was successful (code 200). 


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2018:00:01/04/18', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python>', 'Title': 'How to add a password log in into screen scraping with Beautiful Soup / Python', 'ViewCount': 471, 'id': 48089391, 'user_AboutMe': None, 'user_AccountId': 12575377.0, 'user_DisplayName': 'Kyle', 'user_Reputation': 11.0, 'user_Views': 7.0, 'user_id': 9147959.0}}