Muti-hot encoding vs Label-Encoding I am learning about different input-vector representations for Neural Networks

One of the alternatives to sparse One-Hot encoded vector is the Multi-Hot encoding.

Do I understand correctly that a traditional binary approach to counting numbers is exactly what the Multi-Hot is?  We can imagine a byte as a vector of 8 components, and each entry is either 0 or 1.0

Second part of the question:

I would like to use Multi-Hot encoding to express up to 255 possible input values. If I use the binary-approach, will this be identical to Label-encoding? In a sense that network will figure out that 00000010 is "superior" to 00000001, that there is a "strong correlation and precedence"?

Or is it less exaggerated? For example, in Label-encoding, I could merely use 1 input neuron, just with a varied strength of 0 to 255, much like an enum. There the effect would be really exaggerated, as it presents a really obvious precedence of value $A$ over value $A-1$

This means I can't use it for 255 distinct categories (or for relatively unrelated categories)

Is the effect as bad in Multi-Hot encoding, in particular in a binary approach?



If it's the same as Label-Encoding (which only uses 1 input neuron), why would people ever consider Multi-hot, bloating the dimension of the input-vector?

This post points out we can use multi-hot when the input should contain $N$ concatenated one-hot vectors. For example to represent $N$ entities each of which can belong to $Z$ distinct categories.  Is there also another use?
You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into the model (category 1 &lt; category 2 &lt; category 3....) . Binary encoding introduces false additive relationships between the categories (e.g. category 4 + category 1 = category 5 or 100 + 001 = 101) but fewer of them.

Therefore, binary will usually work better than label encoding, however only one-hot encoding will usually preserve the full information in the data. 

Unless your algorithm (or computing power) is limited in the number of categories it can handle, one-hot encoding will be preferred over other encoding schemes. If you are limited, mean encoding is a powerful alternative because it transform a categorical feature into a numeric one (giving you the minimal number of inputs) while preserving the most important information in the data.

[Mean encoding replaces every category with its target mean. The mean encodings need to be constructed carefully on a separate dataset to avoid data leakage. If you want to reduce your input dimensions as much as possible, this can be helpful. Mean encoding can also be helpful as an additional feature and is very popular on Kaggle to squeeze out some extra performance. This is also known as target encoding or likelihood encoding.]