Split dictionary of lists into two dicts based on binary values I have a very large dictionary that contains key-value pairs where the key is a string and the value a list. All lists have the same lengths. I want to split up the dataset (dictionary) in two based on the values in one of the lists. One of the lists will contain binary values (i.e. 0 or 1). I want to split up the dictionaries so that one of them contains all those where the binary value is 0 and the other where it is 1.

(It is easier to understand this when you mentally transform the dataset in a table where each index of the lists is a row, and where the dictionary values are column names.)

I guess I could first convert the dict to a Pandas dataframe and then subset the dataframe, and convert the resulting dataframes back to dicts, but I only want to do this if it proves to be faster than doing it with dictionaries only.

The following code works (test it here), but as said I'm not sure about efficiency. I'm only interested in Python >= 3.6

from collections import defaultdict

def split_d(d, key):
  include = defaultdict(list)
  exclude = defaultdict(list)

  for i in range(0, len(d[key])):
    binary = d[key][i]
    for k, v in d.items():
      if k == key:
        continue

      if binary == 0:
        exclude[k].append(v[i])
      elif binary == 1:
        include[k].append(v[i])
      else:
        raise ValueError(f"Key {key} is not binary. Expected values 0 or 1")

  return include, exclude

d = {
  'prof': [1,0,0,1,1,1,1,1,0,1],
  'val': [45,12,36,48,48,59,5,4,32,7]
}

print(split_d(d, 'prof'))
You should indent with 4 spaces, as per PEP8.

You can move your if out of the inner loop, and use enumerate, leading to a ~9% speed up.

def split_d(d, key):
    include = defaultdict(list)
    exclude = defaultdict(list)
    for i, binary in enumerate(d[key]):
        if binary == 1:
            output = include
        elif binary == 0:
            output = exclude
        else:
            raise ValueError(f"Key {key} is not binary. Expected values 0 or 1")

        for k, v in d.items():
            if k == key:
                continue
            output[k].append(v[i])

    return include, exclude


You can also remove key from d and add it back when you've gone through both the loops. Leading to a further ~5% speed up.

def split_d(d, key):
    include = defaultdict(list)
    exclude = defaultdict(list)
    keys = d.pop(key)
    try:
        for i, binary in enumerate(keys):
            if binary == 1:
                output = include
            elif not binary:
                output = exclude
            else:
                raise ValueError(f"Key {key} is not binary. Expected values 0 or 1")

            for k, v in d.items():
                output[k].append(v[i])
    finally:
        d[key] = keys
    return include, exclude


Other than the above your code looks good and I don't think there is any way to increase the performance in vanilla Python. And so if this is not fast enough then looking into NumPy would be the best thing next.