Financial website scraper I wrote the code that would scrap Finviz for tickers and href that attach to the ticker. I tested with multi-page as well as one page. I'm new to python programming and therefore believe my code is terrible. I'm hoping someone would help me review it and point out ways to improve it. My goal is to have the code more scalable than what I have now.

import bs4 as bs
import requests

class finviz_crawler():
    def requesting(self,type):
        items=1
        baseurl='https://finviz.com/screener.ashx?v=111'
        if type == 'sp500':
            firstpage=baseurl+'&amp;f=idx_sp500&amp;o=ticker'
        elif type == "China":
            firstpage=baseurl+'&amp;f=cap_largeover,geo_china'
        finalurl=firstpage
        headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'}
        resp=requests.get(finalurl,headers=headers)
        soup = bs.BeautifulSoup(resp.text,"lxml")
        table=soup.findAll('td',{'class':'screener-body-table-nw'})
        maxpage_a_tag=[]
        maxpage_td=soup.findAll('td',{'class':'body-table'},bgcolor="#ffffff",valign="top",width="100%")
        tickerdict={}
        for tags in maxpage_td[0].findAll('a'):
             maxpage_a_tag.append(tags)
        if not maxpage_a_tag:
            maxpage=1
        elif (maxpage_a_tag[-1].text=="next"):
            maxpage=maxpage_a_tag[-2].text
        else:
            maxpage=maxpage_a_tag[-1].text
        for page in range(int(maxpage)):
            resp=requests.get(finalurl,headers=headers)
            soup = bs.BeautifulSoup(resp.text,"lxml")
            table=soup.findAll('td',{'class':'screener-body-table-nw'})
            for row in table:
                 ticker_rows=row.findAll('a',{'class':'screener-link-primary'},href=True)
                 for tickers in ticker_rows:
                     tickerdict[tickers.text]=tickers['href']
            items+=20
            finalurl="{}{}{}".format(firstpage,"&amp;r=",str(items))
        print(tickerdict)


Call the class using finviz_crawler().requesting('China')
Here a few suggestions (in no particular order) for improvements on your code:

Make variables have locality

The line:

items = 1


is at the top of the method, but is only used by the loop at the bottom of the method.  It is likely better located just in front of the loop that increments it.

Same applies to:

tickerdict = {}


You can break strings across multiple lines:

This:

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'}


Can be:

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) '
    'AppleWebKit/537.36 (KHTML, like Gecko) '
    'Chrome/56.0.2924.87 Safari/537.36'
}


You often don't need loops:

for tags in maxpage_td[0].findAll('a'):
    maxpage_a_tag.append(tags)


likely works just fine as:

maxpage_a_tag.extend(maxpage_td[0].findAll('a'))


And then this:

maxpage_a_tag = []
maxpage_a_tag.extend(maxpage_td[0].findAll('a'))


is really just:

maxpage_a_tag = list(maxpage_td[0].findAll('a'))


Don't use python names

The name type is used by Python.  Suggest instead changing to type_ or better yet, as suggested from comments, something more descriptive, like request_type.

Reworked code:

Here is the reworked code with the above suggestions, and maybe a few others that snuck in.

class finviz_crawler():

    def requesting(self, type_):
        baseurl = 'https://finviz.com/screener.ashx?v=111'
        if type_ == 'sp500':
            firstpage = baseurl + '&amp;f=idx_sp500&amp;o=ticker'
        elif type_ == "China":
            firstpage = baseurl + '&amp;f=cap_largeover,geo_china'
        else:
            raise ValueError("Unknown type {}".format(type_))
        finalurl = firstpage
        headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) '
                          'AppleWebKit/537.36 (KHTML, like Gecko) '
                          'Chrome/56.0.2924.87 Safari/537.36'}
        resp = requests.get(finalurl, headers=headers)
        soup = bs.BeautifulSoup(resp.text, "lxml")
        maxpage_td = soup.findAll('td', {'class': 'body-table'},
                                  bgcolor="#ffffff", valign="top",
                                  width="100%")
        maxpage_a_tag = list(maxpage_td[0].findAll('a'))
        if not maxpage_a_tag:
            maxpage = 1
        elif (maxpage_a_tag[-1].text == "next"):
            maxpage = maxpage_a_tag[-2].text
        else:
            maxpage = maxpage_a_tag[-1].text

        items = 1
        tickerdict = {}
        for page in range(int(maxpage)):
            resp = requests.get(finalurl, headers=headers)
            soup = bs.BeautifulSoup(resp.text, "lxml")
            table = soup.findAll('td', {'class': 'screener-body-table-nw'})
            find_all_args = 'a', {'class': 'screener-link-primary'}
            for row in table:
                tickerdict.update(
                    {tickers.text: tickers['href']
                     for tickers in row.findAll(*find_all_args, href=True)})
                items += 20
            finalurl = "{}{}{}".format(firstpage, "&amp;r=", str(items))

        for data in tickerdict.items():
            print(data)


finviz_crawler().requesting('China')