Creating a list from combinations of the original data I am trying to create a list based on some data, but the code I am using is very slow when I run it on large data. So I suspect I am not using all of the Python power for this task. Is there a more efficient and faster way of doing this in Python?

Here an explanantion of the code:


  You can think of this problem as a list of games (list_type) each with a list of participating teams and the scores for each team in the game (list_xx).For each of the pairs in the current game it first calculate the sum of the differences in score from the previous competitions (win_comp_past_difs); including only the pairs in the current game.  Then it update each pair in the current game with the difference in scores. Using a defaultdict keeps track of the scores for each pair in each game and update this score as each game is played.


In the example below, based on some data, there are for-loops used to create a new variable list_zz.

The data and the for-loop code:

import pandas as pd
import numpy as np
from collections import defaultdict
from itertools import permutations

list_type = [['A', 'B'], ['B'], ['A', 'B', 'C', 'D', 'E'], ['B'], ['A', 'B', 'C'], ['A'], ['B', 'C'], ['A', 'B'], ['C', 'A', 'B'], ['A'], ['B', 'C']]

list_xx = [[1.0, 5.0], [3.0], [2.0, 7.0, 3.0, 1.0, 6.0], [3.0], [5.0, 2.0, 3.0], [1.0], [9.0, 3.0], [2.0, 7.0], [3.0, 6.0, 8.0], [2.0], [7.0, 9.0]]

list_zz= []
#for-loop
wd = defaultdict(float)
for i, x in zip(list_type, list_xx):
    # staff 1
    if len(i) == 1:
        #print('NaN')
        list_zz.append(np.nan)
        continue
    # Pairs and difference generator for current game (i)
    pairs = list(permutations(i, 2))
    dgen = (value[0] - value[1] for value in permutations(x, 2))
    # Sum of differences from previous games incluiding only pair of teams in the current game
    for team, result in zip(i, x):
        win_comp_past_difs = sum(wd[key] for key in pairs if key[0] == team)
        #print(win_comp_past_difs)
        list_zz.append(win_comp_past_difs)
    # Update pair differences for current game
    for pair, diff in zip(pairs, dgen):
        wd[pair] += diff
print(list_zz)



Which looks like this:

[0.0,
 0.0,
 nan,
 -4.0,
 4.0,
 0.0,
 0.0,
 0.0,
 nan,
 -10.0,
 13.0,
 -3.0,
 nan,
 3.0,
 -3.0,
 -6.0,
 6.0,
 -10.0,
 -10.0,
 20.0,
 nan,
 14.0,
 -14.0]


If you could elaborate on the code to make it more efficient and execute faster, I would really appreciate it.
The main way to do this faster is to use fewer loops and to do more in each loop. (AFAICT)

loops in there...


A (for event)
A.1 (make team pairs)
A.2 (make dgen)
A.3 (for team, result in event)
A.3.1.1 (for key in pairs)
A.3.1.2 (sum)
A.4 (for pair diff)


You have a lot more flexibility if you use the for loop instead of mostly comprehensions and permutations. And, permutations is wasteful, here, (as discussed, below) because it prevents a few optimisations.

A key change is we can do A vs B and B vs A in the same pass of the loop. So, we can change our loop from "for each team go through all teams" to "for each team go through all following teams". It's a big improvement.

For the line-up A, B, C, D, our iterations look like:


AB &amp; BA
AC &amp; CA
AD &amp; DA


.


BC &amp; CB
BD &amp; DB


.


CD &amp; DC
(done)


So, the algorithm might end up something like (in pseudo code):

past_score_differences = defaultDict(float)

for each event:

    team_outcomes = defaultDict(float)

    teams_len = len(teams in event)
    for i in range(teams_len - 1):
        home_team = teams[i]
        home_score = scores[i]

        for j in range(i+1, teams_len):
            away_team = teams[j]
            away_score = scores[j]

            team_outcomes[home_team] += past_score_differences[(home_team, away_team)]
            team_outcomes[away_team] += past_score_differences[(away_team, home_team)]

            past_score_differences[(home_team, away_team)] += home_score - away_score
            past_score_differences[(away_team, home_team)] += away_score - home_score

        list_zz.append(team_outcomes[home_team])

    # the last team doesn't go through the outer loop
    list_zz.append(team_outcomes[teams[-1]])


Still a lot of loops, but far fewer and with fewer iterations for the innermost loop.

If there is a lot of consistency in the team IDs, then you might consider not using a tuple as the key but instead using a dict within a dict, e.g.

example_past_differences = {
    'A': {
        'B': 10,
        'C': 11,
        'Etc...': 99
    }
}

# later, when accessing

home_team = ...
home_score = ...
home_past_differences = past_score_differences[home_team]
for ...
    away_past_differences = past_score_differences[away_team]
    ...
    team_outcomes[home_team] += home_past_differences[away_team]
    team_outcomes[away_team] += away_past_differences[home_team]


One last thing, your variable names are not easy to understand. I made mine way too long for accesibility to understand, but generally I think the guidance in these slides is very useful: https://talks.golang.org/2014/names.slide#1