Python Web Scraper Using Requests - Not Redirecting Like It Should
Question: So I was bored, and I decided to do some web scraping just for fun and work on
my programming skills. I tried to scrape a more "difficult" site such as
<http://www.aa.com> (American Airlines). I say difficult because it has a
redirect url after you search for a flight.

My current code is:

    
    
        import requests
        from bs4 import BeautifulSoup
    
        req = self.session.get("http://www.aa.com/homePage.do", allow_redirects=True)
        cookies = req.cookies
        params = {
        "originAirport": "JFK",
        "destinationAirport": "LAX",
        "flightParams.flightDateParams.travelMonth": "3",
        "flightParams.flightDateParams.travelDay": "11",
        "flightParams.flightDateParams.searchTime": "120001",
        "carrierPreference": "F",
        "flightSearch": "revenue",
        "tripType": "oneWay",
        "fromSearchPage": "true",
        "searchCategory": "false",
        "adultPassengerCount": "1",
        "searchType": "matrix"
        }
    
    req = self.session.post("http://www.aa.com/reservation/tripSearchSubmit.do;jsessionid=" + str(cookies.get("JSESSIONID")), data=params, allow_redirects=True);
                soup = BeautifulSoup(req.text)
                print(str(req.history) + "\n" + str(req.url))
    
                print(soup.prettify())
    

But this isn't working correctly. It just takes me to the loading page but
doesn't redirect to the flight fares page (the req.history array is empty).
Anyone have any ideas on what I am doing wrong?


Answer: `requests` will automatically handle HTTP redirects, but not necessarily other
kinds of redirects. In particular, the page you linked does a Javascript
redirect (and potentially other Javascript behavior). Remember, `requests` is
a library which makes HTTP requests, but it does not implement the full range
of behaviors of a proper web browser (most notably Javascript).

You can special case around this by studying the page contents and
implementing behavior which mirrors the redirect.

To handle this correctly in the general case, you need something with more
awareness of how web browsers work. Most commonly, this would be an actual web
browser driven by an automation library, for example Selenium:
<https://pypi.python.org/pypi/selenium>

