Scraping web data with Python 3
I assume is_good_response is just checking for a 200 response code.
Merge is_good_response, get_html_content and the insides of your for-loop in your main together.


This makes the main code:

from requests import get
from bs4 import BeautifulSoup

if __name__ == '__main__':
    baseUrl = 'https://www.bikesales.com.au/'
    url = 'https://www.bikesales.com.au/bikes/?q=Service%3D%5BBikesales%5D'

    content = get_html_content(url)
    html = BeautifulSoup(content, 'html.parser')
    BikeList = html.findAll("a", {"class": "item-link-container"})

    for bike in bike_list:
        individualBikeURL = bike.attrs['href']
        bike_content = get_bike(baseUrl+individualBikeURL)


Where we will be focusing on:

def get_bike(url):
    multiplier = 1
    while (BikeContent == None):
        time.sleep(2*multiplier)
        try:
            with closing(get(url)) as resp:
                content_type = resp.headers['Content-Type'].lower()
                if 200 &lt;= resp.status_code &lt; 300:
                    return resp.content
        except RequestException as e:
            print("Error during requests to {0} : {1}".format(url, str(e)))
        if (multiplier &lt; 16):
            multiplier *= 2
    return None



Allow a retry argument. Retry should also act differently on different values:


None - Don't retry.
-1 - Retry infinatly.
n - Retry until \$2^n\$.
iterator - loop through for the delays


We can also add another function to work the same way your previous code did.
You shouldn't need to use contextlib.closing, as Response.close "should not normally need to be called explicitly."
You don't need content_type in get_bike.
You should use *args and **kwargs so you can use requests.gets arguments if you ever need to.
You can allow this to work with post and other request methods if you take the method as a parameter.




import itertools
import collections.abc

import requests.exceptions


def request(method, retry=None, *args, **kwargs):
    if retry is None:
        retry = iter()
    elif retry == -1:
        retry = (2**i for i in itertools.count())
    elif isinstance(retry, int):
        retry = (2**i for i in range(retry))
    elif isinstance(retry, collections.abc.Iterable):
        pass
    else:
        raise ValueError('Unknown retry {retry}'.format(retry=retry))

    for sleep in itertools.chain([0], retry):
        if sleep:
            time.sleep(sleep)
        try:
            resp = method(*args, **kwargs)
            if 200 &lt;= resp.status_code &lt; 300:
                return resp.content
        except requests.exceptions.RequestException as e:
            print('Error during requests to {0} : {1}'.format(url, str(e)))
    return None


def bike_retrys():
    for i in range(5):
        yield 2**i
    while True:
        yield 16




To improve the rest of the code:


Use snake case.
Constants should be in upper snake case.
Use the above code.
Use import requests, rather than from requests import get.
You can make a little helper function to call request, so usage is cleaner.




import requests
from bs4 import BeautifulSoup


def get_bike(*args, **kwargs):
    return request(requests.get, bike_retrys(), *args, **kwargs)


if __name__ == '__main__':
    BASE_URL = 'https://www.bikesales.com.au/'
    url = 'https://www.bikesales.com.au/bikes/?q=Service%3D%5BBikesales%5D'

    content = get_bike(url)
    html = BeautifulSoup(content, 'html.parser')
    bike_list = html.findAll("a", {"class": "item-link-container"})

    for bike in bike_list:
        bike_content = get_bike(BASE_URL + bike.attrs['href'])