How do I parallely check for existence of an item across multiple lists in Python?
Question: As a part of my project, for every word in the dictionary d (shown in the
example code snippet below), I need to check for its existence across
different lists `f1, f2, f3`. I have shown only 3 lists here. And based on the
occurrence, I need to calculate two output values(rule inputs and weights).
The problem I am facing here is, the word can occur in any number of lists,
say word1 in dict d occurs in lists `f1, f2, f3` (shown below) and word2 in
dict d occurs in f1 and f2 and word3 occurs only in one list f3. I have 100s
of these individual lists. I need an efficient and straight forward method to
calculate output values(rule inputs and weights) for every word in dictionary
d based on their varying occurrences across these lists, so that I don't have
to check for every combination of occurrence and write a separate condition
for it, which would make things complicated and ugly.

**P.S.:** The lists are of different size. In example below, f1, f2 and f3 are
of different sizes.

**My Code:**

    
    
    import itertools
    
    d = {'Rosa': 0.023, 'code': 0.356, 'Syntel': 0.144, 'Robotics': 0.245, 'Web': .134, 'sanskrit': 0.23, 'Tamil': 0.23}
    f1 = [['Syntel', 0.2, 4, 0.46, 7, 0.9], ['code', 0.45, 9, 0.43, 2, 0.23], ['Robotics', .43, 3, .1, 3, .73]]
    f2 = [['Web', 0.5, 5, 0.06, 6, 0.9], ['code', 0.05, 1, 0.28, 2, 0.73]]
    f3 = [['Web', 0.5, 5, 0.06, 6, 0.9], ['sanskrit', 0.05, 1, 0.28, 2, 0.73], ['Tamil', 0.23, 4, .13, 5, .23], ['code', 0.32, 4, 0.12, 4, .24]]
    
    # specific case where I am checking if a word of the dictionary occurs in all of the lists f1, f2 and f3
    # I have to write chunk of code for every possible combo of occurrence which I think is a bad approach
    # I am brain stuck ! Help please !!
    for word, score in d.iteritems():
        for x in f1:
            if word == x[0]:
                for y in f2:
                    if word == y[0]:
                        for z in f3:
                            if word == z[0]:
                                A = x[2] * x[3]
                                B = x[4] * x[5]
                                C = y[2] * y[3] + 1
                                D = y[4] * y[5] + 1
                                E = z[2] * z[3] + 1
                                F = z[4] * z[5] + 1
                                mfs = [[A, B], [C, D], [E, F]]
                                weights = sum([x[3], x[5], y[3], y[5], z[3], z[5]])
                                rule_inputs = list(itertools.product(*mfs))
                                len_comb = len(rule_inputs)
                                # 6 --> need code to find this automatically
                                weight_factor = (len(mfs) * len_comb) / 6
                                weights *= weight_factor
                                rule_inputs = sum([sum(r) for r in rule_inputs])
                                print word, rule_inputs, weights
    


Answer: As Joel Cornett says, you probably should be using `dict`s rather than `list`s
in the first place.

But if you need the `list`s for some reason… well, if you're going to search
through a `list` multiple times, you probably want to build a `dict` to search
through:

    
    
    d1 = {elem[0]: elem for elem in f1}
    

Then, instead of this:

    
    
    for z in f3:
        if word == z[0]:
    

… you can just do this:

    
    
    z = d3.get(word)
    if z is not None:
    

You may also want to follow EAFTP and just `try` the whole thing. Your whole
loop then looks like this:

    
    
    for word, score in d.iteritems():
        try:
            x, y, z = d1[word], d2[word], d3[word]
        except KeyError:
            continue
        A = x[2] * x[3]
        # etc.
    

This is assuming you specifically need three lists, as opposed to an arbitrary
number. If you needed to be able to work with any number of lists, you'd do
this:

    
    
    list_of_dicts = [{elem[0]: elem for elem in lst} for lst in list_of_lists]
    for word, score in d.iteritems():
        try:
            values = [d[word] for d in list_of_dicts]
        except KeyError:
            continue
        A = values[0][2] * values[0][3]
        # etc.
    

* * *

There are a few alternatives to this, but this is probably the one you want.

You can `sort` each list and use `bisect` instead of an iterative linear
search, or use something like
[`SortedCollection`](http://code.activestate.com/recipes/577197-sortedcollection/)
to wrap that up for you, or
[`blist.sortedlist`](https://pypi.python.org/pypi/blist/) for a similar type.
This makes the search O(log N) instead of O(N), and makes the code simpler.
But a `dict` makes the search O(1) instead of O(N), and makes the code exactly
as simple as using a sorted list, so, unless you're dealing with keys that are
not hashable (and you're not), why bother?

You can also wrap up the `for`/`if` by writing a `find_in_list` function,
which gives you the same simplicity as a `dict` or `sortedlist`, but without
the performance gains. This could be useful if the keys are neither hashable
nor orderable, or if you had a huge number of tiny lists (so tiny that a
linear search is actually faster than a dict or tree lookup—maybe around size
2-3?). But otherwise, you're just doing extra work (to write the
`find_in_list` wrapper) to slow yourself down, so again, why bother?

