partial fitting, how to ensure one hot captures all features consistently
If you're using sklearn, this is a great use of CountVectorizer as a workaround since you can specify a vocabulary.

To start, get a list of all the 1000 categories and set that as the vocabulary in the transformer. Then convert the column to a string data type and apply this transformer to each batch:

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(vocabulary = pre_made_category_list)
endoded_variable_matrix = cv.fit_transform(dfX[categorical_column])


Even though it's technically a count vectorizor, since there is only one word in the string, the counts for each row will be 1 for the category it is and 0 for everything else, so its effectively one hot encoding the variable. The order of the columns will be the order of the vocabulary, so the matrix will be consistent between folds.