title: Domain-shift Conditioning using Adaptable Filtering via Hierarchical Embeddings for Robust Chinese Spell Check
authors: Minh Nguyen Gia H. Ngo Nancy F. Chen
text: 
  
  
  
  
  Journal of  Class Files, Vol. 14, No. 8, August 2021
  Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals
  
  Domain-shift Conditioning using Adaptable Filtering via Hierarchical Embeddings for Robust Chinese Spell Check
      Minh Nguyen Student Member, IEEE,
          Gia H. Ngo,
          and Nancy F. Chen Senior Member, IEEE
  
  © 2021 IEEE.
  Personal use of this material is permitted.
  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
  
  Nancy F. Chen is the corresponding author (nancychen@alum.mit.edu).
  
      
  
  ==============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
  
  Spell check is a useful application which processes noisy human-generated text.
  Spell check for Chinese poses unresolved problems due to the large number of characters, the sparse distribution of errors, and the dearth of resources with sufficient coverage of heterogeneous and shifting error domains.
  For Chinese spell check, filtering using confusion sets narrows the search space and makes finding corrections easier.
  However, most, if not all, confusion sets used to date are fixed and thus do not include new, shifting error domains.
  We propose a scalable adaptable filter that exploits hierarchical character embeddings to (1) obviate the need to handcraft confusion sets, and (2) resolve sparsity problems related to infrequent errors.
  Our approach compares favorably with competitive baselines and obtains SOTA results on the 2014 and 2015 Chinese Spelling Check Bake-off datasets.
  
  
  
  
  
  
  § INTRODUCTION
  
  Spell check is a common task in processing written text, as spell checkers are an integral component in text editors and search engines.
  A spell checker must identify erroneous words/characters and suggest candidates for correction (Figure <ref>).
  Despite its utility, spell check for Chinese presents unresolved challenges stemming from the language's numerous characters (up to 10k), the sparsity of errors, and the scarcity of linguistic resources with sufficient coverage of heterogeneous and shifting error domains.
  
  
  
  
   §.§ Background
  
  Spell check is more challenging for Chinese than for languages like English as there are more options (characters) to correct an erroneous character.
  There are up to 10k common Chinese characters compared to only 26 characters for English.
  The sparse and shifting errors also make training a good spell checker challenging.
  Errors belong to different domains depending on whether the text with errors is typed or hand-written.
  User demographics and degree of formality may also shift the domain of errors.
  For example, a spell checker trained using errors collected from essays hand-written by second language learners for academic exams may not generalize well to text typed by native language users on social media platforms.
  
  
  Figure <ref> illustrates the challenges from three perspectives using datasets from 3 consecutive years of the Chinese spell check (CSC) Bake-offs <cit.>.
  Figure <ref>a shows the histogram of error types in the 3 training sets, in which there is a long tail with more than 2000 error types that occur only once (error sparsity).
  Figure <ref>b shows the numbers of error types that occur in 1, 2, or all 3 datasets.
  The error domains differ vastly between datasets (i.e. domain-shift <cit.>) with only 1% of error types common across all 3 datasets.
  Most types of error occur exclusively in one dataset (83%).
  
  Confusion sets <cit.> are often used to tackle error sparsity.
  The confusion set of a character consists of phonologically or morphologically similar characters.
  For example, the confusion set of UTF8bsmi無 can be {UTF8bsmi吾,嫵,舞}.
  Due to their similarity, UTF8bsmi無 can be mistaken for any character in its confusion set.
  As confusion sets define characters that are likely substitution errors, they can be used to filter out unlikely corrections even for characters unseen in the training data.
  
  However, confusion sets <cit.> are often handcrafted and may be susceptible to domain-shift.
  As error domains in the data diverge from error domains captured by the handcrafted confusion sets, the number of errors covered by the confusion sets drop to 73%, leaving a quarter of the errors at risk of mis-detection (Figure <ref>c).
  The reduced coverage may also limit the ability to detect infrequent spelling errors, which is in reality a sizable amount of the errors (more than 60% of errors only occur once), given the extremely long tail in the error distribution (see Figure <ref>).
  
  Constructing confusion sets covering all error domains is nontrivial because there are numerous different domains with various error patterns.
  Besides the difference between typed and hand-written text, even within typed text, error patterns differ depending on the input method (IME) used.
  Using phonetic-based IMEs like Sogou Pinyin requires inputting pronunciation, so errors often are caused by pronunciation similarity.
  In contrast, using morphology-based IMEs like Wubi requires inputting characters' components, so errors are often caused by morphological similarity.
  Covering multiple domains is required for accurate spell check because even the same user may commit errors in different domains when using different IMEs (e.g. a user who typically uses morphology-based IMEs might switch to phonetic-based IMEs when she forgets how to write but knows how to pronounce a Chinese character).
  
  Keeping the confusion sets up-to-date is as difficult as constructing them as error patterns evolve or new domains emerge.
  There is much work attempting to expand or construct confusion sets to increase coverage <cit.>.
  However, these methods involve human experts handcrafting the function that measures similarity between characters <cit.> or handcrafting how similar characters are gathered <cit.>.
  It is difficult to scale methods with handcrafted rules to capture shifting error domains.
  
  
  
   §.§ Proposed Solution
  
  Due to the sparsity of spelling errors (many errors observed only once in the training data, or possibly not at all) and the large number of Chinese characters, training a spell check model that directly predicts spell error corrections is difficult.
  We propose a spell check model with an adaptable filter that refines its prediction by considering phonological and morphological similarity between characters.
  The adaptable filter is constructed using a similarity function learned from data using hierarchical character embeddings <cit.>.
  Since similarity between characters is one of the primary factors causing substitution errors <cit.>, the refinement would make spell check more accurate.
  Although character embeddings have been used as model input in prior work <cit.>, as far as we know, our work is the first that uses character embeddings to filter model output to improve accuracy.
  Unlike handcrafted confusion sets, our filter can be trained using errors in the training data so it is less affected by domain-shift.
  Experimental results show that the proposed model with adaptive filtering is more accurate than baseline models, obtaining SOTA results on the 2014 and 2015 Chinese Spelling Check Bake-off datasets.
  
  
  
  § RELATED WORK
  
  Spell check involves error detection and correction to which there are several approaches.
  Error detection was tackled using language model <cit.>, conditional random field <cit.>, graph-based algorithm <cit.>, or sequence tagging model <cit.>.
  
  Most approaches to error correction <cit.> share two common points.
  First, they use confusion sets <cit.> to filter out unlikely correction candidates or to generate candidates for beam-search decoding <cit.>.
  The confusion sets are often constructed once using hand-crafted similarity functions <cit.> and stay fixed thereafter.
  Second, most approaches involve data augmentation <cit.> or transfer learning <cit.> to compensate for the limited size of CSC training sets.
  
  Our work is similar to <cit.>, although our filtering is different.
  Xie et al. <cit.> used fixed filtering while our filtering is adaptable.
  Hong et al. <cit.> filtering can be fine-tuned albeit manually while our filter can be fine-tuned automatically using training data.
  While Cheng et al. <cit.>'s filtering exploits only similarity between characters, our approach also exploits character structural similarity by using hierarchical embeddings.
  Our filter adaptation is a form of supervised domain adaptation <cit.> in which labeled errors help capture additional error domains that are missed by the given confusion sets.
  Although domain adaptation is usually framed as semi-supervised learning where target domain examples are unlabeled <cit.>, semi-supervised adaptation has yet to outperform supervised adaptation in terms of performance improvement <cit.>.
  Thus, through adaptation, our approach may generalize to more error domains and scale more elegantly since it does not require manual tuning.
  
  
  
  § APPROACH
  
  The model consists of a pre-trained masked language model (BERT <cit.>) and our proposed Hierarchical Embedding Adaptable filter (HeadFilt).
  Leveraging pre-trained language model (LM) has led to successes in numerous NLP tasks <cit.>.
  However, predicting corrections using only the masked LM (Figure <ref>a) is challenging due to inadequate training data and the large number of Chinese characters.
  To boost accuracy, LM's predictions are filtered using HeadFilt (Figure <ref>b).
  
  
  Let the input sequence be 𝐗_*, where X_i denotes the i-th character in the sequence. All unique characters across all sequences belong to the vocabulary {X}.
  Given 𝐗_*, the masked LM outputs Ỹ_*, whereby Ỹ_i={Ỹ_ik} is the distribution over corrections at i.
  The predicted correct character at i before filtering is Z̃_i (Equation <ref>).
  
  For each input character X_i, HeadFilt computes a similarity vector of size N, 𝐒_i, which is the similarity between X_i and the set of all N characters in the vocabulary (i.e. {X}).
  HeadFilt is described in Section <ref>.
  The similarity vector 𝐒_i is element-wise multiplied with Ỹ_i to filter out unlikely corrections.
  The predicted correct character at position i after filtering is Z̃^filt_i (Equation <ref>).
  
      Ỹ_*    = MaskLM (𝐗_*) 
      Z̃_i    = _kỸ_ik
      𝐒_i    = HeadFilt (X_i; {X}) 
      Z̃^filt_i    = _k (Ỹ_i⊙𝐒_i)_k
  
  
  
  
   §.§ Adaptable Filter with Hierarchical Embeddings
  
  Filtering out unlikely candidates using confusion sets can significantly boost accuracy <cit.>.
  However, under domain-shift, fixed confusion sets could miss errors from new domains.
  We propose an adaptable filtering model named HeadFilt which can be fine-tuned using errors from new domains to alleviate the domain-shift problem.
  Filter fine-tuning is a form of supervised domain adaptation as it minimizes the domain-shift between errors captured by the given confusion sets and the errors in the training data.
  HeadFilt estimates likelihood of correction candidates using distances in the embedding space between characters. 
  HeadFilt uses hierarchical character embeddings <cit.> which have inductive bias of the character structure so as to extend filtering to phonologically and morphologically characters unseen in the training data, thus addressing the error sparsity problem.
  Hierarchical character embedding is obtained by applying TreeLSTM <cit.> on the tree structure of the character.
  Figure <ref> shows the tree structure of a character and its hierarchical embedding (𝐡_7).
  
  
  A (fixed) confusion set of a character can be represented as a binary similarity vector (𝐒 in Figure <ref>a) with a 1 for each character in the set that is deemed possibly confused with the given character.
  In contrast, the similarity vector (𝐒 in Figure <ref>c) produced by HeadFilt has real-valued scores estimated using the structure of the characters.
  Combining all the confusion sets (i.e. concatenating all the binary vectors) results in a similarity matrix (Figure <ref>b).
  
  
  Let the hierarchical embeddings of character a and b be 𝐡_𝐚 and 𝐡_𝐛.
  The HeadFilt similarity between a and b, S(a, b), is estimated using Equation <ref>.
  The constants β and m in Equation <ref> are the scaling factor and the margin respectively.
  Intuitively, S(a, b) in Equation <ref> is close to 1 if the L2 distance (d_ab) between the two embedding vectors is smaller than the margin m and is close to 0 otherwise.
  Let 𝐒_i be the HeadFilt similarity vector of the character X_i.
  Vector 𝐒_i is the concatenation of similarity scores of X_i and all other N characters (c_1, c_2, … c_N) in the vocabulary. (Equation <ref>).
  
      d_ab   = ‖𝐡_𝐚 / ‖𝐡_𝐚‖ - 𝐡_𝐛 / ‖𝐡_𝐛‖‖
      S(a, b)    = 1/1 + exp(β× (d_ab - m))
      𝐒_i    = [S(X_i, c_1), …S(X_i, c_N)]
  
  
  In addition, let 𝐒_i be the confusion set similarity vector of X_i. The prediction after filtering (Z̃^filt_i) is:
  
      Ỹ^filt_i    = Ỹ_i ⊙𝐒_i   (using fixed confusion set)
      Ỹ^filt_i    = Ỹ_i ⊙𝐒_i   (using HeadFilt)
  
  
  
  
   §.§ Adaptable Filter Training
  
  HeadFilt is trained by minimizing the contrastive loss <cit.> in Equation <ref> below.
  Minimizing the loss forces L2 distances between similar characters to be within the margin m and L2 distances between dissimilar characters to be greater than the margin m.
  Let S(a, b) be the observed similarity of two characters a and b.
  
      L_c    = ∑_a, b [ S(a, b) ×max (0, d_ab - m) 
         + (1 - S(a, b)) ×max (0, m - d_ab) ]
  
  
  There are two sources of examples for training the filter: i.e. the positive and negative examples from the confusion sets and the positive examples observed in the training data.
  A positive example is an error pair (a, b) where a is mistaken for b or vice-versa.
  Thus, all pair (a, b) in the same confusion set and all errors (a, b) in the training data are positive examples.
  Other character pairs are considered as negative examples.
  For positive examples, a and b are similar and S(a, b) = 1, while for negative examples, a and b are dissimilar and S(a, b) = 0.
  
  The objective of the optimization in Equation <ref> is to ensure that (1) the distances between similar characters are smaller than the margin m and (2) the distances between dissimilar characters are larger than the margin m.
  When a and b are similar, S(a, b) = 1, the loss contributed by a and b is max(0, d_ab – m).
  Minimizing the loss will push the embeddings of a and b closer to one another if only d_ab > m.
  When a and b are dissimilar, S(a, b) = 0, the loss contributed by a and b is max(0, m - d_ab).
  Minimizing the loss will push the embeddings of a and b further apart if only d_ab < m.
  
  HeadFilt is trained in two steps:
  (1) imitating the given confusion sets, where characters in the same confusion sets are cast as positive examples and characters not in the same confusion sets are cast as negative examples.
  (2) domain-shift conditioning (adaptation), where error pairs observed in the training data are further added as positive examples and the filter is trained further using this larger set of examples.
  Combining examples from both source and target domains for supervised adaptation was also done in <cit.>.
  The effect of domain adaptation is shown in Section <ref>.
  
  To calculate S(a, b) (Equation <ref>), we need the values of m and β.
  In our experiments, we set m = 0.4.
  Besides, we need to set β so that when a and b are dissimilar, the probability of predicting b as the correction for a or vice-versa is very small (less than chance) as shown in Equation <ref>.
  
      1/1 + exp(β× (d_ab - m))≤1/N , 
         ∀ a, b   s.t.   S(a, b) = 0 
         β≥ln(N - 1)/d_* - m
         d_* = ∑_a,b (1 - S(a, b)) * d_ab/∑_a,b (1 - S(a, b))
  
  However, solving Equation <ref> for all pairs (a, b) (including unobserved pairs) is intractable.
  Thus, we approximate Equation <ref> by Equation <ref> using positive and negative examples in the training data.
  Setting β according to Equation <ref> ensures that on average when a and b are dissimilar, it is very unlikely to predict b as the correction for a.
  
  
  
  § EXPERIMENTS
  
  
  
   §.§ Data
  
  We used datasets from the 2013 <cit.>, 2014 <cit.>, and 2015 <cit.> Chinese Spell Check Bake-offs.
  For each dataset, we only used its training set to fine-tune the models (BERT and HeadFilt).
  Table <ref> shows the statistics of the datasets.
  Unlike 2014 and 2015, there are two test sets in 2013: one for error detection and one for error correction.
  
  
  We used the confusion sets <cit.> provided by the 2013 Bake-off for filtering in our baseline model and for training the HeadFilt model.
  As the confusion sets were created in 2011, it may not cover newer error domains (i.e. domain-shift).
  
  Table <ref> shows the number of errors in the training data that are covered by the confusion sets.
  Although coverage is above 90% for 2013, it drops by 20% absolute for 2014 and 2015.
  This drop could conceivably be due to different data collection methods.
  While the 2013 dataset was from written essays by first language learners of Chinese, the 2014 and 2015 datasets were from typed essays by second language learners of Chinese.
  Despite the lack of coverage, HeadFilt can adapt to errors in the training sets to filter better.
  
  
  
   §.§ Baselines
  
  The proposed approach has two stages in which the masked LM's output from the first stage is filtered by HeadFilt in the second stage.
  We compared the proposed approach against 3 baselines:
  
      
    1. BERT which does not filter predictions.
      
    2. FixedFilt which uses the provided fixed confusion sets for filtering.
      
    3. GlyceFilt which uses Glyce embeddings <cit.> instead of hierarchical embeddings.
  
  Although there are many other embedding models for Chinese characters <cit.>, we chose the Glyce embedding model as it is the current SOTA for many different Chinese NLP tasks and is therefore a strong baseline.
  Results of winning teams of the Bake-offs are also included:
   <cit.> (SMT),
   <cit.> (n-gram LM),
  National Kaohsiung University of Applied Sciences (KUAS),
   <cit.> (HanSpeller),
  as well as recent results by <cit.> (DPL-Corr), <cit.> (FASpell), and <cit.> (MERT).
  
  Note that Wang et al. <cit.> and Cheng et al. <cit.> used different training and test sets, different evaluation tools, and Simplified Chinese characters instead of the original Traditional Chinese.
  Therefore their results are not directly comparable to previously mentioned studies.
  We include our best attempt to compare with <cit.> in Section <ref>.
  
  
  
  
  
  
   §.§ Experimental Setup
  
  We used tools provided by the CSC Bake-offs <cit.> to evaluate accuracy, precision, recall, and F1 score.
  Metrics are calculated at the sentence level, i.e. a prediction is correct if all errors in the sentence are predicted correctly.
  
  The models are implemented using PyTorch <cit.>.
  BERT model is trained using cross-entropy loss for 20 epochs using a learning rate of 3e^-5 and a batch size of 16 with AdamW <cit.>.
  We repeated this training 5 times using 5 different random seeds and reported the average.
  HeadFilt is trained using the contrastive loss <cit.> with Adam <cit.>.
  We used a learning rate of 3e^-3 and a batch size of 500.
  The hierarchical embeddings' dimension is 512.
  HeadFilt is trained in two steps (see Section <ref>).
  HeadFilt is first trained to imitate the given confusion sets for 150k steps.
  HeadFilt is then further trained using the additional error pairs observed in the training data for 50k steps.
  
  
  
   §.§ Results for Traditional Chinese Data
  
  For error detection (Table <ref>), approaches with filtering (FixedFilt, GlyceFilt, and HeadFilt) achieved higher precision and lower recall when compared to the non-filtering baseline (BERT).
  HeadFilt was consistently better than the BERT baseline in terms of the accuracy and F1 score, validating the benefit of filtering.
  HeadFilt was also consistently better than FixedFilt, showing that adaptation is beneficial.
  HeadFilt was as good as if not better than GlyceFilt, suggesting that the inductive bias of hierarchical embeddings is suitable for filtering.
  The proposed HeadFilt model also achieves SOTA results for error prediction F1 score for 2014 and 2015. 
  
  For error correction (Table <ref>), HeadFilt is better than FixedFilt and BERT in all datasets in both accuracy and F1 score.
  Whereas filtering lowers recall in error prediction, it leads to higher recall in error correction since filtering lowers the number of correction candidates to be considered.
  HeadFilt achieved consistently higher recall than the FixedFilt baseline.
  HeadFilt is also better than the SOTA result for F1 score in 2014 and 2015.
  
  
  
   §.§ Results for Simplified Chinese Data
  
  
  
  Due to differences in the experimental setup, results from Wang et al. <cit.> and Cheng et al. <cit.> are not directly comparable to those from prior studies <cit.>.
  First, datasets were converted from Traditional to Simplified Chinese in <cit.>, potentially altering the results.
  This is because distinct traditional characters could be merged and replaced with the same Simplified character, so a character may be correct in Simplified Chinese but erroneous in Traditional Chinese <cit.>.
  For example, the character UTF8bsmi周 in UTF8bsmi周末 (“weekend”) is correct in Simplified Chinese, but it is an error in Traditional Chinese (the correct character is UTF8bsmi週).
  Second, Wang et al. <cit.> evaluated only on sentences with errors from the original test sets using character level metrics instead of evaluating all the test set sentences using sentence level metrics as it was done at the Bake-offs.
  Since we are following the standard set up from the Bake-offs, it is tricky to compare fairly with <cit.>.
  
  
  
  However, it is possible to compare with Cheng et al. <cit.>'s result since they also evaluated using sentence level metrics.
  We attempted to compare against Cheng et al. <cit.>'s method (SpellGCN) by training our models with their training data and evaluated on their test data (in Simplified Chinese).
  The results in Table <ref> and <ref> show that HeadFilt outperforms SpellGCN in all three years.
  HeadFilt was also generally better than GlyceFilt.
  Interestingly, the FixedFilt baseline is worse than the baseline without filtering (i.e. BERT).
  The confusion sets used in the FixedFilt baseline are obtained by converting the original confusion sets into Simplified script.
  The drop in performance of FixedFilt could be because the confusion sets in simplified scripts do not capture the errors well.
  In this setup using simplified characters, only 60% of the errors in the training set are covered by the confusion sets as compared to more than 75% in the setup using traditional characters (Table <ref>).
  Such a large shift in error domains severely affects model performance of FixedFilt.
  In contrast, with domain adaptation, the HeadFilt model still performs well.
  Comparing Table <ref> and <ref> with Table <ref> and <ref> shows how model performance varies across different experimental setups.
  
  
  
   §.§ Ablation Study
  
  We perform ablation to gauge the contribution of components to the performance of HeadFilt.
  We compared the full HeadFilt model against the HeadFilt model without domain adaptation in Section <ref>.
  This is denoted as “-Ad” in Table <ref> and <ref>.
  We further assessed the contribution from the hierarchical embeddings in Section <ref> by replacing hierarchical embeddings with standard embeddings.
  This is denoted as “-Ad -HE”.
  
  
  
    §.§.§ Adaptation Analysis
  
  Domain adaptation contributed little to HeadFilt performance in 2013 since removing this step resulted in the same accuracy and F1 score for both error prediction and correction.
  This is not surprising considering that 93.68% of the errors in the training data is already covered by the confusion sets (shown in Table <ref>), therefore, adaptation only add 17 new pairs of positive examples.
  For 2014 and 2015, domain adaptation shows more tangible contribution, especially for error correction.
  This result aligns with statistics in Table <ref>, which shows larger domain-shifts in 2014 and 2015.
  
  
  
    §.§.§ Hierarchical Embedding Analysis
  
  Besides domain adaptation, the use of hierarchical embeddings also contributed to the performance gain since replacing hierarchical with standard embeddings leads to worse performance for both error prediction and correction.
  This trend is observed in all 3 datasets.
  Besides, HeadFilt without domain adaptation (“-Ad”) consistently beats FixedFilt, demonstrating the benefit of hierarchical embeddings.
  Hierarchical embeddings impose additional constraints (inductive bias) on the embedding space, such that characters with similar morphology are more likely to have similar character embeddings.
  This inductive bias could make it more effective to automatically learn which sets of characters are more likely to be mistaken from one another (See Section <ref>).
  
  
  
  
  
  § DISCUSSION
  
  
  
   §.§ Hierarchical Embeddings Enables Automated Domain Adaptation
  
  Section <ref> shows that using confusion sets without sufficient coverage could hurt performance.
  Ideally, there should be a confusion set of at least 100 candidates for every character <cit.>.
  The confusion sets used in this work <cit.> cover only 5,401 high frequency Traditional Chinese characters and only 30% of the confusion sets have more than 100 candidates.
  Filtering using these fixed confusion sets could miss many spelling errors, especially for text from other domains (e.g. Simplified Chinese text or technical documents).
  Changes in word usage might also make confusion sets obsolete since past spelling errors could become widely accepted and used and are no longer considered as errors.
  For example, in English, “ingot” was a spelling error of the word “lingot”.
  However, as the usage of “ingot” spread, “ingot” became the dominant spelling and is considered a correct word nowadays.
  
  HeadFilt is applicable to text in different domains or to text written by different demographic groups because of its adaptability.
  Theoretically, one could construct confusion sets automatically <cit.> for all characters.
  In practice, it is difficult to construct ones that work well across different input methods (typed and written), different demographics (including new populations with specific slangs and vocabulary) and different domains.
  Constructed confusion sets eventually get outdated overtime, so domain adaptation approaches are useful for a number of reasons.
  
  Domain adaptation for filters requires estimating similarity between Chinese characters, which is algorithmically challenging because Chinese characters are not sequences of sub-character units; they have hierarchical structures.
  Previous work estimated similarity between characters using handcrafted formulas based on edit distance <cit.>, Dice coefficients <cit.>, or overlap in constituent structure <cit.>.
  Edit distance was originally formulated for sequences while Dice coefficient was originally formulated for sets, so they might not capture the hierarchical nature of Chinese characters well.
  Besides, since these measures are handcrafted, they likely require manual fine-tuning to work well.
  In contrast, hierarchical embeddings can better characterize the similarity between Chinese characters and enable automated domain adaptation, reducing the manual effort to keep the filters up-to-date.
  
  
  
  
  
   §.§ Better Recall with HeadFilt
  
  The way hierarchical embeddings are optimized (Figure <ref>) could conceivably explain why HeadFilt achieves higher error correction recall than filtering using given confusion sets (FixedFilt) (Section <ref> and <ref>).
  
  As UTF8bsmi吾 is a substitution error of UTF8bsmi無, minimizing the contrastive loss increases their similarity and reduces the L2 distance between their embeddings.
  Since UTF8bsmi無 and UTF8bsmi舞 have similar morphological structures, their hierarchical embeddings would be close in the embedding space.
  Thus, increasing the similarity between UTF8bsmi吾 and UTF8bsmi無 would indirectly increase the similarity between UTF8bsmi吾 and UTF8bsmi舞.
  Even though UTF8bsmi吾 and UTF8bsmi舞 are not observed as substitution errors, by leveraging the morphological similarity, HeadFilt can still infer that there might be a substitution error.
  Since many errors are not observed during training, using hierarchical embeddings widens the filter's coverage.
  
  
  
   §.§ Future Work
  
  Although HeadFilt models characters' similarity using their structures, which implicitly encode pronunciation <cit.>, there may still be ambiguous cases.
  Therefore, incorporating Pinyin or Zhuyin into the proposed HeadFilt could be a possible extension that further enhances similarity estimation and consequently improves spell checking.
  This is an interesting possibility for future work, as the design and construction of such a study would require one to explicitly consider the contexts of the various different phonetic and tonal representations of a heteronym (a heteronym is a Chinese character that can take on multiple pronunciations), which requires extensive linguistic expertise and resources.
  Such efforts are out of the scope of this current endeavor.
  
  Although this work focuses on spell check for Chinese, it can be extended to languages with similar logographic and phonological structures such as Japanese, Korean, or Vietnamese <cit.>.
  Besides spell check, other tasks such as automatic speech recognition may also achieve higher recall by using HeadFilt.
  There are many homophones in Chinese languages and predicting the correct characters from the audio signal is at times challenging.
  HeadFilt can narrow the list of candidate characters with similar pronunciation to lower the perplexity for speech recognition.
  
  
  
  § CONCLUSION
  
  We presented a filtering model using hierarchical character embeddings (HeadFilt) for Chinese spell check.
  Our approach achieved SOTA error correction results on two spell check datasets.
  The adaptability of HeadFilt makes it possible to tailor proposed model to text from different domains, text written using different input methods, and potentially text written by different individuals.
  
  
  
  IEEEtran
  
  
  