Get all node descendants in a tree
id

you shadow the builtin id with this name as variable

itertuples

a way to improve performance is using itertuples to iterate over the DataFrame: for _, node, parent in df.itertuples():

iterations

You can do this in 1 iteration over the input with a nested loop over the ancestors:

node_ancestors = defaultdict(set)
node_ancestors[0] = set([0])
node_descendants = defaultdict(set)
node_descendants[0] = set([0])
for _, node, parent in df.itertuples():
    node_ancestors[node].add(node)
    node_ancestors[node].update(node_ancestors[parent])

    for ancestor in node_ancestors[node]:
        node_descendants[ancestor].add(node)


Depending on how nested the tree is, this will be faster or slower than iterating over the whole input twice. You'll need to test it on your dataset.

global vs local

another speedup might be achieved by doing this in a function instead of the global namespace (explanation)

def parse_tree(df):
    node_ancestors = defaultdict(set)
    node_ancestors[0] = set([0])
    node_descendants = defaultdict(set)
    node_descendants[0] = set([0])
    for _, node, parent in df.itertuples():
        node_ancestors[node].add(node)
        node_ancestors[node].update(node_ancestors[parent])

        for ancestor in node_ancestors[node]:
            node_descendants[ancestor].add(node)

    return node_ancestors, node_descendants