why is issubset knowingly reducing one of my columns
embarrassed to admit that it took wayyy tooo looonnnggg to bring my mind at peace.
issubset is taking EACH VALUE within x41 column and comparing in xtrain and xvalid. the thing is, x41 has 17000 unique values, so obviously when doing the split, the values will not be in BOTH sets. Thus, it is NOT a subset, because not all xvalid['x41'] values are in xtrain['x41'].
phew. life makes sense again.
(apologies about formatting, this is my first question, and when finalizing question, there was no back button?)
my categorical columns that i KNOW are categorical and even object_cols CONFIRMS this are:
'x34', 'x35', 'x41', 'x45', 'x68', 'x93'. All 6 of them are present in xtrain and xvalid. But why is x41 being kicked out by the issubset operation? Even though we can clearly see x41 is present in object_cols_val
why is the issubset messing up and throwing out x41?
what is this doing:
[col for col in object_cols if set(xvalid[col]).issubset(set(xtrain[col]))]
I thought it's checking each column from object_cols in xvalid, then checking to see if it's a subset of xtrain, WHICH IT IS. Ugh, why is it treating x41 differently? (probably not related but x41 has a $ and numbers, but why would that matter? as long as the column is present in both sets?)
all categorical columns
object_cols_train=[col for col in xtrain.columns if xtrain[col].dtype =='object']
print("object_cols are:",object_cols)
object_cols_val=[col for col in xvalid.columns if xvalid[col].dtype =='object']
print("object_cols_val are:",object_cols_in_val)
"good" columns that can safely be ordinal encoded
good_label_cols=[col for col in object_cols if set(xvalid[col]).issubset(set(xtrain[col]))]
print("good_label_cols are:",good_label_cols)
"bad" problematic columns that should be dropped (for now, but i believe we should NEVER drop)
bad_label_cols=list(set(object_cols)-set(good_label_cols))
print("bad_label_cols are:",bad_label_cols)

outputs:


object_cols are: ['x34', 'x35', 'x41', 'x45', 'x68', 'x93']
object_cols_val are: ['x34', 'x35', 'x41', 'x45', 'x68', 'x93']
good_label_cols are: ['x34', 'x35', 'x45', 'x68', 'x93']
bad_label_cols are: ['x41']

I'm still beginner/intermediate, i tried separating out the sets to see what they look like, but cant because 'col'.
I tried:
xtrain[col]
set(xtrain[col])
set(xvalid[col]).issubset(set(xtrain[col]))
I KNOW what xtrain['x41'] and xvalid['x41'] look like.
Maybe i should include here:
xtrain['x41'].head(),xvalid['x41'].head()

output:

(22449    $-996.73
39178    $-361.51
33715      $851.5
36010    $-765.51
13370    $-1391.9
Name: x41, dtype: object,
34320     $412.48
27355    $-473.03
18144    $-208.31
20740    $-434.41
10805     $203.53
Name: x41, dtype: object)