tracking frequency of words in an ebay search result
Question: using python 3.5, what im looking to do is to go to the results page of an
ebay search by means of generating a link, save the source code as an xml
document, and iterate thru every individual listing, of which there could be
1000 or more. next i want to create a dictionary with every word that appears
in every listing's title, (title only) and its corresponding frequency of
appearance. so for example, if i search 'honda civic', and the thirty of the
results are 'honda civic ignition switch', i'd like my results to come out as
`results = {'honda':70, 'civic':60, 'igntion':30, 'switch':30, 'jdm':15,
'interior':5}` etc, etc.

heres a link i use: [http://www.ebay.com/sch/Car-Truck-
Parts-/6030/i.html?_from=R40&LH_ItemCondition=4&LH_Complete=1&LH_Sold=1&_mPrRngCbx=1&_udlo=100&_udhi=700&_nkw=honda+%281990%2C+1991%2C+1992%2C+1993%2C+1994%2C+1995%2C+1996%2C+1997%2C+1998%2C+1999%2C+2000%2C+2001%2C+2002%2C+2003%2C+2004%2C+2005%29&_sop=16](http://www.ebay.com/sch/Car-
Truck-
Parts-/6030/i.html?_from=R40&LH_ItemCondition=4&LH_Complete=1&LH_Sold=1&_mPrRngCbx=1&_udlo=100&_udhi=700&_nkw=honda+%281990%2C+1991%2C+1992%2C+1993%2C+1994%2C+1995%2C+1996%2C+1997%2C+1998%2C+1999%2C+2000%2C+2001%2C+2002%2C+2003%2C+2004%2C+2005%29&_sop=16)

the problem im having is that i only get the first 50 results, instead of the
X,000's of results i potentially will get with different search options. what
might be a better method of going about this?

and my code:

    
    
    import requests
    from bs4 import BeautifulSoup
    from collections import Counter
    
    r = requests.get(url)
    myfile = 'c:/users/' + myquery
    fw = open(myfile + '.xml', 'w')
    
    soup = BeautifulSoup(r.content, 'lxml')
    for item in soup.find_all('ul',{'class':'ListViewInner'}):
        fw.write(str(item))
    fw.close()
    print('...complete')
    
    fr = open(myfile + '.xml', 'r')
    wordfreq = Counter()
    for i in fr:
        words = i.split()
        for i in words:
            wordfreq[str(i)] = wordfreq[str(i)] + 1
    
    fw2 = open(myfile + '_2.xml', 'w')
    fw2.write(str(wordfreq))
    fw2.close() 
    


Answer: You are getting the first 50 results because EBay display 50 results for each
page. The solution is to parse one page at time. With this search, you can use
a different url:

[http://www.ebay.com/sch/Car-Truck-
Parts-/6030/i.html?_from=R40&LH_ItemCondition=4&LH_Complete=1&LH_Sold=1&_mPrRngCbx=1&_udlo=100&_udhi=700&_sop=16&_nkw=honda+%281990%2C+1991%2C+1992%2C+1993%2C+1994%2C+1995%2C+1996%2C+1997%2C+1998%2C+1999%2C+2000%2C+2001%2C+2002%2C+2003%2C+2004%2C+2005%29&_pgn=1&_skc=50&rt=nc](http://www.ebay.com/sch/Car-
Truck-
Parts-/6030/i.html?_from=R40&LH_ItemCondition=4&LH_Complete=1&LH_Sold=1&_mPrRngCbx=1&_udlo=100&_udhi=700&_sop=16&_nkw=honda+%281990%2C+1991%2C+1992%2C+1993%2C+1994%2C+1995%2C+1996%2C+1997%2C+1998%2C+1999%2C+2000%2C+2001%2C+2002%2C+2003%2C+2004%2C+2005%29&_pgn=1&_skc=50&rt=nc)

Notice a parameter `_pgn=1` in the url? This is the number of the page
currently displayed. If you provide a number that exceeds the number of the
pages for the search, a error message will appear in a div with class `"sm-
md"`

So you can do something like:

    
    
    page = 1
    url = """http://www.ebay.com/sch/Car-Truck-Parts-/6030/i.html?_from=R40&LH_ItemCondition=4&LH_Complete=1&LH_Sold=1&_mPrRngCbx=1&_udlo=100&_udhi=700&_sop
      =16&_nkw=honda+%281990%2C+1991%2C+1992%2C+1993%2C+1994%2C+1995%2C+1996%2C+
      1997%2C+1998%2C+1999%2C+2000%2C+2001%2C+2002%2C+2003%2C+2004%2C+2005%29&
      _pgn="""+str(page)+"&_skc=50&rt=nc"
    
    has_page = True
    
    myfile = 'c:/users/' + myquery
    fw = open(myfile + '.xml', 'w')
    
    while has_page:
        r = requests.get(url)
        soup = BeautifulSoup(r.content, "lxml")
        error_msg = soup.find_all('p', {'class':"sm-md"})
        if len(error_msg) > 0:
            has_page = False
            continue
        for item in soup.find_all('ul',{'class':'ListViewInner'}):
            fw.write(str(item))
            page+=1
    
    fw.close()
    

I only tested entering the pages and printing the ul, and it worked nice

