\section{Introduction}
Chinese spelling correction (CSC) is a task to detect and correct spelling mistakes in Chinese sentences. 
It has always been an important Chinese NLP task and also gradually becomes an indispensable sub-module for many downstream tasks, 
such as search query correction \citep{JianfengGao2010ALS}, named entity recognition \citep{wan2011named}, and machine translation \citep{zheng2020opportunistic}.

Further exploring the process of input, unlike alphabetic languages, 
the input of Chinese characters must rely on an input method (or \textbf{i}nput \textbf{m}ethod \textbf{e}ditor, commonly abbreviated IME \footnote{https://en.wikipedia.org/wiki/Input\_method}).
Among all Chinese IMEs, pinyin IME \footnote{https://en.wikipedia.org/wiki/Pinyin\_input\_method} is the most popular one, 
used by more than \textbf{97\%} of Chinese people \citep{chen2000new, zhang2021correcting}.
We can infer that most spelling errors are made by people when they input through pinyin IME, such as typing the wrong pinyin, clicking the wrong words, etc.
It will be more practical and valuable to pay attention to this scenario.
Therefore, this paper focuses on spelling errors generated by pinyin IME.

\begin{CJK*}{UTF8}{gbsn}
\begin{figure}[t!]
  \centering
  \includegraphics[width=0.5\textwidth]{./image/problem.jpg}
  \caption{It is hard to repoduce the sampled spelling error from SIGHAN (misspelling “错误” as “错勿”) by pinyin IME, no matter what input form is used.}
  \label{Fig:problem}
\end{figure}
\end{CJK*}

However, there is still no professional benchmark dataset for errors generated by pinyin IME.
In this situation, previous research all use SIGHAN13-15 \citep{ShihHungWu2013ChineseSC, LiangChihYu2014OverviewOS, YuenHsienTseng2015IntroductionTS} as baseline datasets, 
but they cannot accurately evaluate the real performances of CSC systems.
(1) The main reason is that errors in SIGHAN datasets come from mistakes in essays written by teenage students (SIGHAN13) or Chinese as a foreign language (CFL) learners (SIGHAN14-15), 
which has a big gap with errors generated by native speakers through pinyin IME. 
For example, as shown in Figure \ref{Fig:problem}, many errors in SIGHAN would hardly occur if people use pinyin IME 
because the modern pinyin IME always has a strong language model and would not recommend very unreasonable candidate words.
(2) In addition, sentences in SIGHAN datasets are in traditional Chinese, and even after converting to Simplified Chinese, there is still a big difference in the expression.
(3) The data size is too small, for example, SIGHAN13 has only 1,000 test samples, which may make the evaluation results unreliable.


\begin{CJK*}{UTF8}{gbsn}
  \begin{figure}[t!]
    \centering
    \includegraphics[width=0.45\textwidth]{./image/example.jpg}
    \caption{A real post of official media in Sina Weibo, where the “效力于” is misspelled as "效力与".}
    \label{Fig:example}
  \end{figure}
\end{CJK*}

Therefore, in the paper, we first present a \textbf{C}hinese \textbf{S}pelling \textbf{C}orrection \textbf{D}ataset for errors generated by pinyin \textbf{IME} (CSCD-IME).
It contains 40,000 manually labeled data, which is ten times larger than SIGHAN datasets. 
As far as we know, this is also the largest dataset for the CSC task.
As shown in Figure \ref{Fig:example}, sentences in the dataset are all real posts of official media (such as Sina Sports) on Sina Weibo and all in Simplified Chinese.
We could believe that these errors are generated by editors in the process of typing through pinyin IME, which is consistent with errors made by native speakers in the real input scenario.
In order to study the error distribution in depth, we design a tagging system at the pinyin and semantic levels.
The comparative analysis shows that the error distribution of CSCD-IME is quite different from existing datasets, 
with a much higher proportion of word-level errors.

In previous research, additional pseudo data is usually used to improve the performance of CSC systems, 
which is often constructed by ASR based methods \citep{wang2018hybrid} or confusion set based methods \citep{liu2021plome, zhang2021correcting}.
However, these construction methods still have a gap with the real Chinese input scene. 
Therefore, we propose a novel pseudo-data construction method based on pinyin IME.
Specifically, we simulate the input process through pinyin IME and add sampled noise to generate pseudo data.
In this way, pseudo data can be constructed at scale and high quality.

In the end, we experiment with several strong CSC baseline models and perform a series analysis. 
Experiments show that CMCD-IME is a challenging dataset 
and our proposed pseudo-data construction method can better fit the real error distribution than previous methods and improve the performance of CSC systems. 
We also discover how to better use pseudo data, 
and provide a useful strategy in the appendix, including the data scale, the data source, and the training strategy.

\begin{table*}
 
  \centering
 
  \begin{tabular}{cccccccc}
  \hline
  \textbf{Dataset} & \textbf{train size} & \textbf{dev size} & \textbf{test size} & \textbf{all size} & \textbf{avg len.} & \textbf{err. ratio} & \textbf{avg err./sent.} \\
  \hline
  SIGHAN13 & 700 & - & 1,000 & 1,700 & 60.94 & 77.11\% & 1.20 \\ 
  SIGHAN14 & 3,437 & - & 1,062 & 4,499 & 49.66 & 86.19\% & 1.52 \\ 
  SIGHAN15 & 2,339 & - & 1,100 & 3,439 & 31.10 & 81.82\% & 1.33 \\ 
  \hline
  CSCD-IME & 30,000 & 5,000 & 5,000 & \textbf{40,000} & 57.42 & 46.02\% & 1.09 \\ 
  \hline
\end{tabular}
\caption{Data statistics, including the size of the training set, the size of development set, the size of the test set, the size of the all dataset, 
the averaged number of characters per sentence, the ratio of error sentences, and the averaged error characters in each error sentence.}
\label{Tab:statistic}
\end{table*}

\section{Related Work}

\textbf{CSC Dataset}: 
CSC datasets can be divided into two categories according to the user group: CFL learners and native speakers.
Among them, the former has richer data resources, including SIGHAN14-15 \citep{LiangChihYu2014OverviewOS, YuenHsienTseng2015IntroductionTS}, NLPCC2018 \citep{YuanyuanZhao2018OverviewOT}, NLPTEA2020 \citep{GaoqiRao2020OverviewON} and YACLC \citep{wang2021yaclc}. 
Although the latter has a broader application scenario, the related data sources are fewer, only SIGHAN13 \citep{ShihHungWu2013ChineseSC} and CTC2021 \citep{zhao2022overview}. 
Moreover, these two datasets are very small, with almost 1,000 test sentences.
In addition, SIGHAN13 is a traditional Chinese dataset, and CTC2021 is a Chinese grammar correction dataset. 
These are all different from the real scenario of Simplified Chinese spelling error correction.

\textbf{CSC Data Augmentation}: In order to make up for the lack of labeled data, previous studies usually build additional pseudo data to increase the performance.
The mainstream method is based on the confusion set \citep{liu2021plome, zhang2021correcting, zhang2020spelling}, 
the pseudo data constructed in this way is extensive in size but relatively low in quality because of the big gap from the true error distribution.
Another relatively high-quality construction method is based on ASR or OCR method \citep{wang2018hybrid}.
However, this method requires additional labeled ASR or OCR data and is inconsistent with the pinyin IME based input scenario.

\textbf{CSC models}: In recent years, BERT-style \citep{devlin2018bert} models have dominated the research of the CSC task 
\citep{hong2019faspell, liu2022craspell, zhu2022mdcspell, huang2021phmospell,zhang2020spelling, bao2020chunk, liu2021plome}. 
However, due to the lack of large-scale and high-quality datasets, the performance of these models is greatly limited.


\section{CSCD-IME}
In this section, we will show how to build the CSCD-IME and discover the error distribution.

\subsection{Data Collection}
Our goal is to build a dataset that fits the real Chinese spelling error correction scenario where most spelling errors are generated by pinyin IME.
We cleverly discovered a large-scale data source with this feature: LCSTS \citep{hu2015lcsts}, which is a large-scale Chinese short text summarization dataset.
Sentences in this dataset all come from posts of popular Chinese media on Sina Weibo, such as People’s Daily.
These accounts are verified on Weibo and labeled with a blue ‘V’. There are usually special editors responsible for editing and publishing news and information.
We can infer that these editors generally use pinyin IME to publish posts, so spelling errors in this editing progress are generated by pinyin IME, which is consistent with our target scenario.
In addition, LCSTS is a large-scale and general dataset, including over 2 million posts and covering multiple domains such as politics, economics, military, movies, games, etc.
Therefore, we use LCSTS as our data source.

\begin{CJK*}{UTF8}{gbsn}
  \begin{table*}
    \centering
   
   
    \begin{tabular}{l|l|p{10cm}}
      \hline
      \multicolumn{2}{c|}{\textbf{Tags}} & \multicolumn{1}{c}{\textbf{Description}} \\
      \hline
      \multirow{10}{*}{Pinyin} & same pinyin & 
      \tabincell{l}{the pinyin of the correct is the same with of the wrong \\ e.g. 现\textcolor{red}{再}(zai) -> 现\textcolor{blue}{在}(zai); \textit{now} } \\
      \cline{2-3}
      & fuzzy pinyin &  
      \tabincell{l}{the pinyin of the correct is the fuzzy pinyin of the wrong \\ e.g. 现\textcolor{red}{宅}(\textcolor{red}{zh}ai) -> 现\textcolor{blue}{在}(\textcolor{blue}{z}ai); \textit{now} }\\ 
      \cline{2-3}
      & similar pinyin & 
      \tabincell{l}{the edit distance of two pinyin is 1 \\ e.g. 现\textcolor{red}{砸}(za) -> 现\textcolor{blue}{在}(za\textcolor{blue}{i}); \textit{now} } \\
      \cline{2-3}
      & dissimilar pinyin & 
      \tabincell{l}{the edit distance of two pinyin is greater than 1 \\ e.g. 现\textcolor{red}{太}(\textcolor{red}{t}ai) -> 现\textcolor{blue}{在}(\textcolor{blue}{z}ai); \textit{now} } \\
      \hline
      \multirow{10}{*}{Semantic} & entity word & 
      \tabincell{l}{the correct word is a named entity, such as a place name \\ e.g. \textcolor{red}{卢}(lu)山 -> \textcolor{blue}{庐}(lu)山; \textit{Lushan Mountain}} \\
      \cline{2-3}
      & normal word & 
      \tabincell{l}{the wrong word is a valid Chinese word\\ e.g. 开心地\textcolor{red}{驴友}(lv, you) -> 开心地\textcolor{blue}{旅游}(lv, you); \textit{travel happily}} \\
      \cline{2-3}
      & special char & 
      \tabincell{l}{the error is a Chinese high-frequency error-prone character\\ e.g. 开心\textcolor{red}{的}(de)旅游 -> 开心\textcolor{blue}{地}(de)旅游; \textit{travel happily}} \\
      \cline{2-3}
      & normal char & 
      \tabincell{l}{the wrong word is not a valid Chinese word or the length is 1\\ e.g. 开心地旅\textcolor{red}{有}(you) -> 开心地旅\textcolor{blue}{游}(you); \textit{travel happily}} \\
      \hline
    \end{tabular}
    \caption{The tagging system at pnyin level and semantic level.}
    \label{Tab:tag_system}
  \end{table*}
\end{CJK*}

\subsection{Data Selection}
\label{section:selection}

We split posts in LCSTS into sentence levels and obtain over 8 million sentences.
Obviously, it is not realistic to manually label all these sentences, and most of the sentences are completely correct,
so we use an error detection model and rules to filter out these correct sentences.

\textbf{Detection Model}: 
Given a source sequence ${\rm \bf X} = (x_{w_{1}}, x_{w_{2}}, ..., x_{w_{N}})$, the detection model is to check whether a character $w_{i}(1 \leq  i \leq N)$ is correct or not. 
We use the label 1 and 0 to mark the misspelled character and the correct character, respectively.
The detection model can be formalized as follows:
\begin{equation}
  {\rm \bf y_{d}} = sigmoid(W^{T}(E({\rm \bf e})))
\end{equation}
where ${\rm \bf e} = (e_{w_1}, e_{w_2}, ..., e_{w_N})$ is the word embedding and ${E(*)}$ is the pre-trained encoder. 
The output ${\rm \bf y_{d}} = (y_{d_{1}}, y_{d_2}, ..., y_{d_N})$ is a probability sequence, 
and $y_{d_{i}} \in (0,1)$ means the probability that character $x_{w_i}$ is erroneous. 

\textbf{Training}: We follow the successful experience \citep{wang2020combining} of the NLPTEA2020 task \citep{rao2020overview} 
and use a Chinese ELECTRA-Large discriminator model \footnote{https://github.com/ymcui/Chinese-ELECTRA} \citep{clark2020electra} to initialize the detection model. 
Following previous research, we train the detection model on SIGHAN13-15's training data and Wang's pseudo data \citep{wang2018hybrid}
and save the best checkpoint by SIGHAN13-15's test data.

\textbf{Filtering}: We then use the trained detection model to filter out correct sentences. 
For the input sentence, we can obtain the error probability of each character ${\rm \bf y_{d}} = (y_{d_{1}}, y_{d_2}, ..., y_{d_N})$.
\begin{CJK*}{UTF8}{gbsn}
Previous studies have shown that the detection model will not work well on special high-frequency characters (such as “的”, “地”, “得” etc.) 
because of the poor labeling of these characters in SIGHAN \citep{xu2021read}, 
and is also easy to over-check low-frequency entity words (such as person names, place names, etc.) \citep{zhang2020spelling}.
Therefore, we use a Chinese lexical analysis tool (LAC) \citep{jiao2018LAC} to detect these special and entity characters in the input sentence 
and divide characters into three categories: $C_{speicial}, C_{entity}, C_{normal}$.
\end{CJK*} 
We then calculate the maximum error probability of characters in each category (if the category is empty, the maximum probability is 0), and 
only if all the max error probabilities of each category are less than the corresponding threshold is the sentence considered correct.
This can be formalized as follows:
\begin{equation}
  \left\{
\begin{aligned}
  & max(\{y_{d_{i}} | w_{i} \in C_{special}\}) < \delta_{special} \\
  & max(\{y_{d_{i}} | w_{i} \in C_{entity}\}) < \delta_{entity} \\
  & max(\{y_{d_{i}} | w_{i} \in C_{normal}\}) < \delta_{normal}
\end{aligned}
\right.
\end{equation}
where $\delta_{special}$, $\delta_{entity}$ and $\delta_{normal}$ are thresholds.

We use the above method to filter out about 91.2\% of the sentences and keep about 700,000 sentences that may have spelling errors. 
We randomly select 2,000 filtered sentences to check whether these sentences are really correct, 
and the result shows that the accuracy is 99.2\%, which is in line with our expectations.
For the rest of about 700,000 sentences, we cumulatively selected about 50,000 random sentences for manual annotation.

\subsection{Data Annotation}
We recruit a group of educated native speakers for data annotation, and each sentence will be annotated twice by different people to ensure the annotation quality.
Specifically, the annotator needs to detect and correct spelling errors in the sentence and give the final correct sentence.

In order to further clear the labeling rules and reduce objections, 
sentences that may be ambiguous \citep{zhang2022mucgec} will be discarded directly. 
There are three scenarios:
(1) The sentence is very unsmooth, and the annotator does not understand the meaning of the sentence; 
(2) There are multiple possible correction results for the typo in the sentence, and no unique correction result with a pinyin edit distance less than 2 could be found;
(3) The sentence contains complex grammatical errors.
Therefore, the sentence that is not discarded during the labeling process is semantically clear and has a unique correction result.

In the end, we obtain 40,000 labeled and qualified sentences, which make up the \textbf{CSCD-IME} dataset.
The whole dataset is randomly divided into the training set, development set, and test set according to the ratio of 6:1:1.

\begin{figure*}[htbp]
	\centering
	\includegraphics[width=0.95\textwidth]{./image/pinyin.png}
	\includegraphics[width=0.95\textwidth]{./image/semantic.png}
	\caption{The comparison of error distribution (\%) at pinyin level (above) and semantic level (below). If the proportion is less than 2\%, the specific value is not indicated in the figure.}
	\label{fig:analysis}
\end{figure*}

\subsection{Basic Statistics Analysis}
\label{section:statistic}
As shown in Table \ref{Tab:statistic}, the data size of CSCD-IME is much larger than existing datasets.
In terms of the averaged sentence length and averaged errors in sentences, CSCD-IME is closer to SIGHAN13, 
with a longer sentence length and fewer errors because both SIGHAN13 and CSCD-IME are datasets for native speakers, whereas SIGGHAN14 and SIGHAN15 are datasets for CFL learners. 
As for the proportion of positive and negative samples, CSCD-IME is more balanced.

\subsection{Error Distribution Analysis}
\label{section:analysis}
In order to analyze the error distribution of spelling mistakes produced by pinyin IME in depth, 
we first design a tagging system at pinyin level and semantic level, 
and then conduct the error distribution study on CSCD-IME and previous datasets.

\textbf{Tag defination}: As shown in Table \ref{Tab:tag_system}, we design four pinyin-level and four semantic-level tags.
For pinyin-level tags, we define different tags according to the degree of pinyin similarity between the wrong and the correct character. 
In practice, we use pypinyin\footnote{https://github.com/mozillazg/python-pinyin} to obtain the pinyin info and add the pinyin-level tag. 
Note that for \textit{fuzzy pnyin}, we follow the definition of Sogou IME \footnote{https://pinyin.sogou.com/help.php?list=3\&q=5}.
As for semantic-level tags, they can be roughly divided into two categories: 
word-level tags (\textit{entity word} and \textit{normal word}) and character-level tags (\textit{special char} and \textit{normal char}).
In practice, we first seg the corrected sentence by LAC \citep{jiao2018LAC} and get the corresponding word of the wrong character. 
For a given word, we then add the semantic tag based on LAC's POS tag and an entity dictionary.
\begin{CJK*}{UTF8}{gbsn}
Note that the \textit{special char} refers to “他/她/它/的/地/得”.
\end{CJK*}

\textbf{Pinyin-level analysis}: As shown in the upper part of Figure \ref{fig:analysis}, 
in all four datasets, the \textit{same pinyin} error is the largest proportion and the \textit{dissimilar pinyin} error is the smallest one. 
Compared with SIGHAN datasets, this feature of CSCD-IME is more prominent, 
it has higher proportion of \textit{same pinyin} errors (up to 82.4\%) and lower percentage of \textit{dissimilar pinyin} errors (only 2.2\%). 
This is because pinyin IME could auto fix the pinyin typoes based on the input context \citep{jia2014joint}.

\textbf{Semantic-level analysis}: As shown in the lower part of Figure \ref{fig:analysis}, 
the proportion of \textit{special char} error in CSCD-IME and SIGHAN13 is lower than that in SIGHAN14 and SIGHAN15, 
this means native speakers could make fewer \textit{special char} errors than CFL learners.
It also can be observed that the percentage of the word-level errors (\textit{entity word} and \textit{normal word}) in CSCD-IME is significantly higher than those in SIGHAN datasets and is nearly doubled.
This is because if we input through the whole word (Figure \ref{Fig:problem}(b)), pinyin IME will generally recommend a valid word rather than a very strange “word”.
Note that \textbf{the high proportion of word-level errors} is the most important feature of CSCD-IME. 
This is also the biggest difference with SIGHAN datasets and makes CSCD-IME challenging.
We will conduct a deep analysis between word-level and character-level errors in Section~\ref{section:exp}.

\begin{figure*}[t!]
  \centering
  \includegraphics[width=0.9\textwidth]{./image/pseudo.jpg}
  \caption{The process of generating pseudo data by pinyin IME (the orange box indicates the selected noise).}
  \label{Fig:pseudo}
\end{figure*}

\section{Data Augmentation}
It is expensive to manually label data for the CSC task \citep{wang2018hybrid}, so how to construct pseudo data has always been a valuable topic.
In this section, we propose a novel method to construct large-scale and high-quality pseudo data.

\subsection{Data Preparation}
The basic principle of pseudo-data construction is to add noise to correct sentences and generate sentences with spelling errors, 
so we need to prepare completely correct sentences first.
In fact, this kind of text data is common on the Internet, such as sentences in Wikipedia or classic books, 
which also ensures that we could build \textbf{large-scale} dataset.
In practice, we better choose the in-domain text that is relevant to the actual application scenario.

\subsection{Noise Simulation}
Given a correct sentence, we use the pinyin IME to simulate the input of this sentence and add noise in the input process to construct pseudo data.

We first sample a pinyin noise $\upsilon_{pinyin}$, a token granularity noise $\upsilon_{token}$, and a number of errors noise $\upsilon_{num}$. 
As discussed in Section~\ref{section:analysis}, 
the $\upsilon_{pinyin}$ could be one of \textit{same pinyin}, \textit{fuzzy pinyin}, \textit{similar pinyin} and \textit{dissimilar pinyin},
the $\upsilon_{token}$ could be one of \textit{word} and \textit{character}, 
and the $\upsilon_{num}$ is a number greater than 0.
For the convenience of expression, we use \textit{token} to represent the Chinese character or word.
We can obtain the distribution of $\upsilon_{pinyin}$, $\upsilon_{token}$, and $\upsilon_{num}$ from the prior knowledge of the scene or the labeled data.

In the generation process, we first determine the number of errors to generate based on the sampled $\upsilon_{num}$. 
For each error, as shown in Figure \ref{Fig:pseudo},
we randomly select a word (Figure \ref{Fig:pseudo}(a-c)) or a character (Figure \ref{Fig:pseudo}(d-e)) from the correct sentence based on the sampled $\upsilon_{token}$.
Then we first type the correct text \textbf{before} the selected token, and
enter correct (Figure \ref{Fig:pseudo}(a)) or wrong pinyin (Figure \ref{Fig:pseudo}(c)) of the selected token based on the sampled $\upsilon_{pinyin}$.
If the first token recommended by pinyin IME is the correct token, we randomly select the second or the third token as noise (Figure \ref{Fig:pseudo}(b)).
If not, we directly select the first token as noise (Figure \ref{Fig:pseudo}(a)).
Finally, we replace the correct token with the noise token in the original sentence.

\subsection{Language Model Filtering}
\label{lm-filter}
It can be observed that the pseudo data generated by this method may also be correct sentences (Figure \ref{Fig:pseudo}(b)), 
so we use an n-gram language model (LM) to perform secondary filtering.
Specifically, we calculate the perplexity (PPL) value of the generated sentence and the original sentence, 
and only when the PPL value is relatively improved by $\delta$ after adding the noise do we consider the generated noise is indeed erroneous.
This can be formalized as follows:
\begin{equation}
  \frac{PPL(noised) - PPL(origin)}{PPL(origin)} > \delta
\end{equation}
In practice, the value of $\delta$ is adjusted by the selected language model.
This additional step ensures that we could build a \textbf{high-quality} dataset.

Finally, based on the correct sentences from the LCSTS dataset (Section~\ref{section:selection}), 
the error distribution from the CSCD-IME dataset (Section~\ref{section:statistic}, \ref{section:analysis}) 
and the Google pinyin IME \footnote{https://en.wikipedia.org/wiki/Google\_Pinyin}, 
we build a large-scale and high-quality pseudo dataset \textit{LCSTS-IME-2M}, including 2 million sentences.

\begin{table*}
  \small 
  \centering
  \bgroup
  \setlength\tabcolsep{0.6em}
  \def1.2{1.2}
  \begin{tabular}{l|ccc|ccc|ccc|ccc}
  \hline
  \multirow{3}{*}{Models} & \multicolumn{6}{c|}{Sentence-level} & \multicolumn{6}{c}{Character-level} \\
  \cline{2-13}
  & \multicolumn{3}{|c|}{Detection} & \multicolumn{3}{|c|}{Correction} & \multicolumn{3}{|c|}{Detection} & \multicolumn{3}{|c}{Correction} \\
  \cline{2-13}
  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\
  \hline
  {\footnotesize BERT}  & 79.16 & 65.83 & 71.88 & 70.55 & 58.66 & 64.06 & 83.00 & 67.01 & 74.15 & 73.59 & 59.41 & 65.75 \\ 
  {\footnotesize +\textit{LCSTS-ime-2M}} & 78.98 & 73.60 & 76.20 & 75.63 & 70.47 & 72.96 & 82.19 & 75.75 & 78.84 & 78.84 & 72.67 & 75.63 \\
  \cline{2-13}
  {\footnotesize +\textit{Wang-cs-271K}} & 79.76 & 59.53 & 68.17 & 74.40 & 55.54 & 63.60 & 83.99 & 59.77 & 69.84 & 77.60 & 55.22 & 64.53 \\
  {\footnotesize +\textit{Wang-asr-271K}} & 78.18 & 64.39 & 70.62 & 74.01 & 60.96 & 66.86 & 81.67 & 65.23 & 72.53 & 77.07 & 61.55 & 68.44 \\
  {\footnotesize +\textit{Wang-ime-271K}} & 79.72 & 64.18 & 71.11 & 76.86 & 61.88 & 68.56 & 82.9 & 65.39 & 73.11 & 79.84 & 62.98 & \textbf{70.41} \\
  \cline{2-13}
  {\footnotesize *\textit{Wang-cs-271K}} & 37.21 & 22.10 & 27.73 & 25.22 & 14.98 & 18.80 & 39.69 & 23.22 & 29.30 & 26.50 & 15.51 & 19.57 \\
  {\footnotesize *\textit{Wang-asr-271K}} & 52.27 & 44.42 & 48.03 & 43.79 & 37.21 & 40.23 & 55.87 & 46.32 & 50.65 & 46.56 & 38.61 & 42.22 \\
  {\footnotesize *\textit{Wang-ime-271K}} & 52.41 & 44.33 & 48.04 & 46.87 & 39.64 & 42.95 & 56.07 & 49.72 & 52.70 & 49.69 & 44.07 & \textbf{46.71} \\
  \hline
  {\footnotesize soft-masked BERT} & 80.87 & 64.78 & 71.94 & 74.42 & 59.62 & 66.20 & \textbf{84.46} & 65.35 & 73.68 & 77.50 & 59.97 & 67.62 \\
  {\footnotesize +\textit{LCSTS-ime-2M}} & 79.19 & \textbf{74.86} & \textbf{76.97} & 75.75 & \textbf{71.60} & 73.62 & 82.39 & \textbf{77.93} & \textbf{80.10} & 78.63 & \textbf{74.37} & 76.44 \\
  \cline{2-13}
  {\footnotesize +\textit{Wang-cs-271K}} & 79.74 & 62.22 & 69.9 & 75.52 & 58.92 & 66.19 & 83.33 & 62.50 & 71.43 & 78.59 & 58.94 & 67.36  \\
  {\footnotesize +\textit{Wang-asr-271K}} & 79.17 & 66.52 & 72.30 & 75.66 & 63.57 & 69.09 & 83.21 & 67.64 & 74.62 & 79.47 & 64.6 & 71.26 \\
  {\footnotesize +\textit{Wang-ime-271K}} & 79.96 & 66.87 & 72.83 & 77.21 & 64.57 & 70.32 & 83.57 & 68.63 & 75.37 & 80.64 & 66.22 & \textbf{72.72}  \\
  \cline{2-13}
  {\footnotesize *\textit{Wang-cs-271K}} & 37.23 & 17.54 & 23.85 & 23.87 & 11.25 & 15.29 & 34.54 & 18.55 & 24.14 & 22.02 & 11.83 & 15.39 \\
  {\footnotesize *\textit{Wang-asr-271K}} & 46.74 & 38.86 & 42.44 & 41.41 & 34.43 & 37.60 & 47.32 & 41.97 & 44.49 & 42.02 & 37.26 & 39.50 \\
  {\footnotesize *\textit{Wang-ime-271K}} & 56.03 & 51.63 & 53.74 & 51.55 & 47.50 & 49.45 & 59.22 & 57.40 & 58.30 & 54.69 & 53.01 & \textbf{53.84} \\
  \hline
  {\footnotesize PLOME} & 79.78 & 57.23 & 66.65 & 78.09 & 56.01 & 65.23 & 83.48 & 57.99 & 68.44 & 81.49 & 56.61 & 66.81 \\ 
  {\footnotesize +\textit{LCSTS-ime-2M}} & \textbf{81.20} & 72.21 & 76.44 & \textbf{79.05} & 70.30 & \textbf{74.42} & 84.21 & 73.81 & 78.67 & \textbf{82.00} & 71.88 & \textbf{76.60} \\
  \hline
\end{tabular}
\egroup
\caption{The performance (\%) of baseline models on CSCD-IME with or without different pseudo datasets.
Note that "+" means pretraining on pseudo data and then finetuning on CSCD-IME's training data, and "*" means no finetuning step, evaluating directly after pretraining on pseudo data.}
\label{Tab:result}
\end{table*}


\begin{table}
  \small 
  \centering
  \def1.2{1.2}
  \begin{tabular}{cccc}
  \hline
  Models & char level & word level & $\Delta$ \\
  \hline
  BERT & 72.82 & 71.07 & -1.75 \\
  soft-masked BERT & 75.09 & 72.71 & -2.38 \\
  PLOME & 77.77 & 72.78 & \textbf{-4.99} \\
  \hline
\end{tabular}
  \caption{The f1-score (\%) of correction at character level on character-level errors and word-level errors, and the decline of the latter compared with the former.}
\label{Tab:word-level}
\end{table}

\section{Experiment}
\label{section:exp}
In this section, we provide the benchmark of CSCD-IME and verify the effectiveness of our pseudo-data construction method.

\subsection{Basic Settings}
The basic experimental setup includes data, evaluation methods, and baseline models.

\textbf{Data}: 
We use CSCD-IME as the basic dataset, including the training set, development set, and test set.
Note that for pseudo data, we conduct two-step training: pretraining and then finetuning, 
using only pseudo data in the pretraining step, and only manually labeled data in the finetuning step.

\textbf{Metrics}: 
We use the f1-score of detection and correction at sentence level and character level as evaluation metrics.
The sentence-level metrics follow the calculation in FASPell \citep{hong2019faspell}.
As for character-level metrics, we \textbf{do not} follow the method in SpellGCN \citep{cheng2020spellgcn}. 
We calculate \textbf{all} characters for correction metric instead of only those been correctly detected characters.

\textbf{Baselines}: We choose the following three strong baseline models.
\textit{BERT} \citep{devlin2018bert} directly fine-tunes the standard masked language model to generate fixed-length corrections. 
\textit{Soft-Masked BERT} \citep{zhang2020spelling} utilizes a detection model to help the correction model learn the right context. 
\textit{PLOME} \citep{liu2021plome} integrates the phonetic and visual features into the pre-trained model, 
and it has an additional pretraining step on a large-scale confusion set based pseudo dataset.

\subsection{CSCD-IME Benchmark}
In this part, we provide basic benchmark performances of CSCD-IME, including two versions: without any pseudo data and with large-scale pseudo data \textit{LCSTS-IME-2M}.

As shown in Table \ref{Tab:result},
if we don't add any pseudo data, the overall f1-score of correction at character level could be only nearly 67\%; 
when introducing 2 million pseudo data, the performance could increase a lot, but the best is still just over 76\%. 
There is still a big room for improvement.

We further analyze the model's performance (without any pseudo data version) on different semantic tags. 
Note that here we only analyze errors with \textit{same pinyin} to reduce the impact of pinyin.

As shown in Table \ref{Tab:word-level}, 
all three models perform worse on word-level errors than on character-level errors, and the maximum gap could reach 5\%.
The iteration of the CSC model in recent years, for example, from \textit{BERT} to \textit{PLOME}, 
reflects in the improvement of character-level errors but no improvement in word-level errors.
For word-level errors, the model should have a strong understanding of context and entities, 
so these errors are more difficult and require further effort.
Compared with existing datasets, the biggest feature of CSCD-IME is the high proportion of word-level errors, 
so from this point of view, CSCD-IME is challenging.

\subsection{Comparition of Pseudo Datasets}
In this subsection, we compare pseudo datasets constructed by different methods. 
Note that \textit{PLOM} already has a pretraining step on large-scale pseudo data, so this baseline is not used here.

We choose existing ASR(/OCR)-based pseudo data \citep{wang2018hybrid} as experiment dataset, including over 271K sentences, abbreviated as \textit{Wang-asr-271K}.
This pseudo dataset is also the most famous one, which is always used to improve performance in previous research.
We further extract correct sentecnes from \textit{Wang-asr-271K} to construct confusion set based pseudo data \textit{Wang-cs-271K} and pinyin IME based pseudo data \textit{Wang-ime-271K}.

As shown in Table \ref{Tab:result}, our proposed pinyin IME based pseudo-data construction method is more effective than all existing methods. 
Compared with \textit{Wang-asr-271K} and \textit{Wang-cs-271K}, adding \textit{Wang-ime-271K} (ours) always obtain the best f1-score on both sentence-level and character-level metrics.
To observe the effect of each method more clearly, if we \textbf{only} use pseudo data in the training progress, 
our method would significantly outperform other methods.
The key to the quality of pseudo data is the error distribution of pseudo data, as shown in Figure \ref{fig:analysis-pseudo}, 
our method can better fit the real scene. More analysis can be checked in Appendix \ref{pseudo-analysis}.

\subsection{Impact of LM Post-Filtering}

\begin{table}
  \small
  \centering
  \def1.2{1.2}
  \begin{tabular}{cccc}
  \hline
  LM threshold ($\delta$) & precision & recall & f1 \\
  \hline
  w/o & 41.17 & 40.66 & 40.91 \\ 
  -20\% & 44.52 & \textbf{49.01} & 46.66 \\ 
  0\% &  49.69 & 44.07 & \textbf{46.71} \\ 
  20\% &  50.64 & 26.46 & 34.76 \\ 
  50\% & \textbf{57.52} & 9.38 & 16.12 \\ 
  \hline
\end{tabular}
\caption{The correction results (\%) at character level for pseudo data with different LM filtering strategies.}
\label{Tab:ngram}
\end{table}

In this part, we investigate the effect of LM filtering in Section \ref{lm-filter}. 
We reconstruct \textit{Wang-ime-271K} and adopt different LM filtering strategies.
We choose the basic \textit{BERT} model to conduct the experiment, 
and \textbf{only} use pseudo data in the training progress to more clearly study the differences. 

As shown in Table \ref{Tab:ngram}, if we do not use LM filtering, more unexcepted noises will be introduced, for example, the generated pseudo data may also be a completely correct sentence.
However, when the threshold is too low, even less than 0, the generated errors will be more confused, and the model would have good recall but poor precision.
In contrast, if the threshold is too high, the generated errors are relatively simple, and the precision is better, but the recall is lower.
Therefore, LM filtering is necessary and we also need to choose a moderate threshold.

\subsection{Error Analysis}
\begin{CJK*}{UTF8}{gbsn}
We detailed analyze errors that can not be correctly handled in \textbf{all} three baseline models and summarize these errors into the following three main types.

The over-correction of expressions with low-frequency \citep{liu2022craspell}. Since the backbone model is BERT and its pre-trained task is masked token recovering, 
these CSC models tend to overcorrect valid low-frequency expressions to more frequent expressions, such as people's names, place names, new Internet words, etc.
For instance, “墨尔本\textcolor{blue}{矩形}体育场” (Melbourne Rectangular Stadium) is a location entity and is over-corrected as “墨尔本\textcolor{red}{巨形}体育场” (Melbourne Mega Stadium).

The missing correction of expressions requires strong contextual reasoning ability.
If the wrong word in the sentence is a high-frequency word, 
the model needs to combine the semantic information of the context to correct it.
For example, for the sentence “几辆车停在原地争执，后面的车辆也无法\textcolor{red}{进程}” (since several cars parked and argued, cars behind could not enter the city), 
where the wrong word “\textcolor{red}{进程}” (process) is a common word, 
but it is wrong to fill in this sentence, based on the context, it should be “\textcolor{blue}{进城}” (enter the city) here.

The failing correction of grammar-related errors.
The CSC task also contains a certain number of grammar-related errors, 
such as the use of function words, auxiliary words, and verb-object fixed collocations, and these baseline models cannot effectively handle such errors.
For example, for the phrase “准确\textcolor{red}{的}传达情绪” (convey emotions accurately), 
the auxiliary word between the adverb and the verb should be “\textcolor{blue}{地}” not “\textcolor{red}{的}” according to Chinese grammar rules.
\end{CJK*}

\section{Conclusion}
In this paper, we focus on the most common mistakes in practical CSC applications, i.e., spelling errors generated by pinyin IME. 
We first build a new dataset CSCD-IME for this field, which is also the largest dataset for the CSC task. 
Furthermore, we propose a novel method to construct large-scale and high-quality pseudo data based on pinyin IME.
A series of analyses and experiments show that CSCD-IME holds a particular error distribution and is a challenging dataset. 
Our proposed pseudo-data construction method can better fit this real error distribution and is more efficient than all previous methods.
We hope this data resource and our discovery will advance follow-up research.

