Sending POST request to AJAX with Python3.4 and urllib
Question: I am trying to scrape line movements from: <http://scores.covers.com/football-
scores-matchups.aspx>

I need to iterate through each Season and Week of using the Calendar provided:

When I inspect the network to see what is getting sent, I see two POST
requests when I change the Season:

(I've combined the above 3 images into 1 since I don't have enough rep to post
more than 1 image)

<http://s29.postimg.org/773muclra/covers_scrape4.jpg>

I have searched and read and hacked away at it all day and have made no
progress. I can recreate the error I'm getting by using Firebug to edit and
resend the post request if I edit the payload to be something unexpected. So I
have a feeling the problem is in the way I'm encoding and sending the data.
I've tried json.dump, utf-8, Content-Type application/x-www..., in every
combination I can think of.

My current code is as follows:

    
    
    import urllib.request
    import json
    import urllib.parse
    class Scraper():
    
        def __init__(self):
            pass
    
        def scrape(self):
            url = 'http://scores.covers.com/ajax/SportsDirect.Controls.LiveScoresControls.ScoresCalendar,SportsDirect.Controls.LiveScoresControls.ashx?_method=changeDay&_session=no'
    
            data = {
                'league': '1',
                'SeasonString': '2012-2013',
                'Year': '1',
                'Month': '1',
                'Day': '1'
            }        
    
            # data = urllib.parse.urlencode(data).encode('utf-8')
            data = json.dumps(data).encode('utf-8')
    
            headers = {
                'Host': 'scores.covers.com',
                'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0',
                'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                'Accept-Language': 'en-US,en;q=0.5',
                'Accept-Encoding': 'gzip, deflate',
                'Referer': 'http://scores.covers.com/football-scores-matchups.aspx',
                'Content-Type': 'application/json',
                'Connection': 'keep-alive',
                'Pragma': 'no-cache',
                'Cache-Control': 'no-cache'
            }
    
            req = urllib.request.Request(url)
            req.data = data
            req.headers = headers
    
            f = urllib.request.urlopen(req)
            print(f.read())
    
            return f
    

Which gives:

    
    
    In [1]: import scraper
    
    In [2]: s = scraper.Scraper()
    
    In [3]: s.scrape()
    b"new Object();r.error = new ajax_error('System.ArgumentException','Object of type \\'System.DBNull\\' cannot be convert
    ed to type \\'System.String\\'.',0)"
    Out[3]: <http.client.HTTPResponse at 0x4d6b650>
    

Thanks in advance.


Answer: use `data="league=1\nSeasonString=2014-2015\nYear=1\nMonth=1\nDay=1"`

the data is not a json type.

