Chinese word segmentation using neural networks Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in Chinese text since native speakers can easily segment words from text. In order to assist language learning tools, it's helpful to have an automatic way to segment words within blocks of text.
Some approaches use dictionary based greedy algorithm methods but are prone to failing due to the common problems that most greedy algorithms present.
I want to try using neural networks, but my question is, How would one go about encoding the characters to the input neurons of the network?
I am not talking about OCR. The characters are already known and encoded in unicode, but how would I present the characters to the inputs of the network?
One method I can imagine is to have the network look at the text in portions of say 100 characters at a time, and have one neuron for each character. But how would I represent the character as a number to the network? Using the unicode integer number value for the character doesn't seem like a good idea.
So, the main question is about how to represent the Chinese characters in your Chinese word segmentation task.
Since effectively these characters are non-ordinal categorical variables, we would represent these as one hot encodings (https://medium.com/@michaeldelsole/what-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179) of dimensionality n = number of unique Chinese characters in your dataset.
Sometimes n can be very large. So in this instance, we typically have an Embedding layer, which can reduce data sparsity and collapse the number of dimensions. (https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding#:~:text=Arguments%20%20%20%20input_dim%20%20%20,the%20embed%20...%20%202%20more%20rows%20)