Proportional sorting of web scrape results If you have a given list of lists what's the best way to sort them proportionally? Is there a better way to do this?

The idea is that in a case of web scraping, you would want to avoid hitting google all the time and want to distribute calls so that they are as even as possible.

Is this a thing that can be done more efficiently than this terribad code that I wrote?

urls = [
        ['http://google.com/search','redirect.com', 0],
        ['http://duckduckgo.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0],
        ['http://bing.com/search','redirect.com', 0],
        ['http://yahoo.com/search','redirect.com', 0],
        ['http://yahoo.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0],
        ['http://bing.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0],
        ['http://yahoo.com/search','redirect.com', 0],
        ['http://altavista.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0],
        ['http://yahoo.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0],
        ['http://altavista.com/search','redirect.com', 0],
        ['http://google.com/search','redirect.com', 0]
    ]

def get_domain( url ):
    domain = ''
    parts = tldextract.extract( url )
    if parts.subdomain != '' and parts.subdomain != 'www':
        domain = parts.subdomain + '.' + parts.domain + '.' + parts.suffix
    else:
        domain = parts.domain + '.' + parts.suffix
    return domain


def proportional_sort( urls ):
    doms = [get_domain(url[0]) for url in urls]
    dom_set = set(doms)


    matrix = {}
    for domain in dom_set:
        sublist = [item for item in urls if get_domain(item[0]) == domain]
        # print sublist
        random.shuffle(sublist)
        if len(sublist) in matrix:
            matrix[len(sublist)][domain] = sublist
        else:
            matrix[len(sublist)] = {}
            matrix[len(sublist)][domain] = sublist

    # print matrix.keys()

    map_vector = []

    for key in matrix:
        map_vector += [key] * key * len(matrix[key])
        map_vector = sorted(map_vector, key=int)

    # print map_vector

    proportionally_sorted = []

    for pos in map_vector:
        if pos in matrix:
            rand_pos = [pos]
            for key in matrix.keys():
                if key % pos == 0:
                    rand_pos.append(key)
            for position in rand_pos:
                rand_keys = matrix[position].keys()
                random.shuffle(rand_keys)
                for key in rand_keys:
                    rand = rand_keys.pop()
                    if matrix[position][rand]:
                        proportionally_sorted.append(matrix[position][rand].pop())
                        break
                    else:
                        continue

    return proportionally_sorted
I don't really understand what your code is supposed to do but I can see some room for improvement so let's go!

Style

Your code looks good and PEP 8 seems to be respected. I find the spacing in the argument lists a bit unusual but he... Also, your functions probably deserves some documentation like docstrings for instance.

Improvements

domain = '' is not needed here.



In :

    random.shuffle(sublist)
    if len(sublist) in matrix:
        matrix[len(sublist)][domain] = sublist
    else:
        matrix[len(sublist)] = {}
        matrix[len(sublist)][domain] = sublist


You don't need to call len over and over again.

    random.shuffle(sublist)
    leng = len(sublist)
    if leng in matrix:
        matrix[leng][domain] = sublist
    else:
        matrix[leng] = {}
        matrix[leng][domain] = sublist


Then, you could remove the duplicated logic :

    random.shuffle(sublist)
    leng = len(sublist)
    if leng not in matrix:
        matrix[leng] = {}
    matrix[leng][domain] = sublist


But the part with the test and the empty dict could be done automatically with defaultdicts.



In :

def proportional_sort( urls ):
    doms = [get_domain(url[0]) for url in urls]
    dom_set = set(doms)

    matrix = {}
    for domain in dom_set:
        sublist = [item for item in urls if get_domain(item[0]) == domain]


it seems like you are using the wrong data structure. You want to iterate over domains only and only once and then process the corresponding urls. It would be better to define a dictionnary mapping domains to list of urls once so that you don't have to call get_domain many times in many places. You have different ways to do this, the best involves setdefault :

dom_dict = {}
for url in urls:
    dom_dict.setdefault(get_domain(url[0])).append(url)


Then, your code becomes (not tested):

matrix = {}
for domain, sublist in dom_dict.iteritems():
    random.shuffle(sublist)
    leng = len(sublist)
    if leng not in matrix:
        matrix[leng] = {}
    matrix[leng][domain] = sublist