are OneHotEncoder and keras To_categorical same? The length of human_vocab is 18377.
The length of input X is 1000

I'm trying to run to_categorical

np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), X)))


Is this the same if i apply:

onehot_encoder = OneHotEncoder(sparse=False)
onehot_encoder.fit_transform(X)


the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?

human_vocab contains the corpus data that X doesnt contain
I'm trying to find the replacement of to_categorical due to memory issues to create hot vector
To answer your question:

the output of onehot_encode is (1000, 9739) shouldn't it be (1000,18377) ?


The OneHotEncoder instance will create a dimension per unique word seen in the training sample. Here you are only showing it 9739 different words at training so it does not need more dimensions to perform one hot encoding.

One way to have it accommodate the entire vocabulary is

onehot_encoder = OneHotEncoder(sparse=False, categories=human_vocab)
onehot_encoder.fit_transform(X)