How can I handle a column with list data? I have a dataset which I processed and created six features:

['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList',
   'totalProducts']


And the target variable is a binary class (gender).

The feature 'productList' is a list:

    df['ProductList'].head()
Out[169]:
0    [13, 25, 113, 13793, 2, 25, 113, 1946, 2, 25, ...
1    [12, 31, 138, 14221, 1, 31, 138, 1979, 1, 31, ...
2                               [13, 23, 127, 8754, 0]
3    [13, 26, 125, 5726, 2, 26, 125, 5727, 2, 26, 1...
4           [12, 23, 119, 14805, 1, 23, 119, 14806, 0]
Name: ProductList, dtype: object


Now, it is obvious that I can't use this feature as it is. How do I handle this feature? I can explode the list and create a row for each list item, but will it serve my purpose?

Update: I applied OHE after exploding the list, and it results in 10k+ columns, which my GCP instance and my computer can't handle; when applying PCA.

PS: There are over 17,000 unique products.
You basically want to create a column for each product bought, as the presence or absence of each in the list is a feature in itself. See Hadley Wickham’s definition of tidy data.

That being said, you seem to have a large number of products. To avoid the course of dimensionality, what I would do is to take your binary bought/not features (or count values might be even more effective if you have that data) and do dimensionality reduction to get a reasonable set of features. Latent Dirichlet Allocation (which comes from topic modelling), PCA, t-SNE, our UMAP are all easy to implement and worth trying. PCA is the least sophisticated and the fastest to run and would be a good baseline.

When you have your smaller list of features, you might want to try using a classifier that’s further selects the most relevant features, like gradient boosted trees.