Categorical variables with multiple entries transformed to entity embedding I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.

Some columns have more than one entry for the same sample, i.e., sample1 for column1 has entry1 for line1 and entry2 for line2. Here's an example:

df = pd.DataFrame({'pat': [1, 2, 3, 3, 3, 3], 'diag_type': ['D', 'OP', 'D', 'D', 'D', 'OP'],
                  'diag': ['D_1', 'OP_1', 'D_1', 'D_3', 'D_4', 'OP_2']})

    pat     diag_type   diag
0   1   D   D_1
1   2   OP  OP_1
2   3   D   D_1
3   3   D   D_3
4   3   D   D_4
5   3   OP  OP_2


Since I have so many categories I would need to do some restructuring, and I decided that entity embedding is the best way to do it. However, I haven't figure out a good way to restructure the dataframe. I thought about doing some sort of dummy coding as exemplified in this stackoverflow post, e.g.:

d.get_dummies(df.set_index('pat')).sum(level=0)
    diag_type_D     diag_type_OP    diag_D_1    diag_D_3    diag_D_4    diag_OP_1   diag_OP_2
pat                             
1   1   0   1   0   0   0   0
2   0   1   0   0   0   1   0
3   3   1   1   1   1   0   1


But then I still get a lot of sparsity and it takes forever. Is there a better way of doing it?
Why don't you try with a different kind of encoding?

One of the most typical encodings, when you have high cardinality, is Target Encoding. You can find an implementation in this library. You can find an explanation of what is it and when to apply it here.