How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?

let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of the categories. ["Man", "Woman", "Diverse"] has no order to it so having one input that represents them all within one dimension makes little sense.

When using cross validation, the dataset often gets split into a lot smaller subsets. These subsets may not hold all categories. When using OHE of sklearn, the input set is used to determine the dimensionality. This can lead to unpredictable column counts of the networks data input. It can also lead to different categories taking different positions in the NN.

How would one process this input to feed to the NN without hard-coding all possible categories and still be able to handle varying numbers of categories in the input set?

Two intuitive ideas that don't work:

• determine the dataset size (after OHE) and set the input size of the NN based off of that: Doesn't work because each CV subset would potentially have a different model and each category position in the dimensions doesn't map to other subsets
• 0 pad to an arbitrarily high (but likely never reached) input size: dirty, because the ordering is still not ensured and if one category is missing, all inputs may be shifted by 1+ positions

Would autoencoders help?

• I am not giving here a solution to fix OHE coding (I have my reasons to hate OHE including the one you mentioned here), instead I suggest looking into Entity Encoding (an idea of learning an Embedding space borrowed from NLP) to encode categorical encoding esp. when developing NN together with numerical features. Have a look at the original paper arxiv.org/abs/1604.06737, or here for blogpost about it: medium.com/@satnalikamayank12/…. I am wrapping up a complete Notebook in Google Colab with a real example soon. Jan 2 '19 at 13:27

A solution is set a maximum length for the number of categories and fill the future categories in order to see them in your data. Hence, that part of the categories which are not seen in the sampled data would be set to zero.

OneHotEncoding, according to me its a part of pre-processing activity. You should save all the pre-processing models that you have used for preparing the training data, so that you can apply these to unseen data for prediction. Otherwise you will always face such type of error as you have mentioned.

Now in order to handle variable categories in OneHotEncoding, as mentioned above by @OMG, always use maximum possibility of values while training your model. So that you can avoid such type of problem for unseen data.

Sample code snippet:

from sklearn.preprocessing import OneHotEncoder
o_h_e = OneHotEncoder(sparse=False, categories=[(['A', 'B', 'C', 'D'])])
encoded_data = o_h_e .fit_transform(data) #where data is your array of data


But sometimes it's not feasible to determined all the possible classes in advance:

1. I think in such case we can keep one type as unknown, where we can put all not defined type as unknown. It may need retraining of model after a certain period.
2. Retrained a model
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.28183606266975403, "perplexity": 1213.3159641061882}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 20, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-43/segments/1634323585177.11/warc/CC-MAIN-20211017113503-20211017143503-00663.warc.gz"}