```latex
\begin{CJK*}{UTF8}{gbsn}
% ADDITION Chinese support
\vspace*{0.35in}

% Title must be 250 characters or less.
% Please capitalize all terms in the title except conjunctions, prepositions, and articles.
\begin{flushleft}
{\Large
\textbf\newline{Optimizing the Learning Order of Chinese Characters Using a Novel Topological Sort Algorithm}
}
\newline
% Insert author names, affiliations and corresponding author email (do not include titles, positions, or degrees).
\\
James C. Loach\textsuperscript{*},
Jinzhao Wang\textsuperscript{}
\\
\bigskip
INPAC and Dept. of Physics, Shanghai Jiao Tong University, Shanghai 200240, China.
\bigskip

% Insert additional author notes using the symbols described below. Insert symbol callouts after author names as necessary.
%
% Remove or comment out the author notes below if they aren't used.
%
% Primary Equal Contribution Note
%\Yinyang These authors contributed equally to this work.

% Additional Equal Contribution Note
% Also use this double-dagger symbol for special authorship notes, such as senior authorship.
%\ddag These authors also contributed equally to this work.

% Current address notes
%\textcurrency a Insert current address of first author with an address update
% \textcurrency b Insert current address of second author with an address update
% \textcurrency c Insert current address of third author with an address update

% Deceased author note
%\dag Deceased

% Group/Consortium Author Note
%\textpilcrow Membership list can be found in the Acknowledgments section.

% Use the asterisk to denote corresponding authorship and provide email address in note below.
* email@example.com

\end{flushleft}
% Please keep the abstract below 300 words
\section*{Abstract}
We present a novel algorithm for optimizing the order in which Chinese characters are learned, one that incorporates the benefits of learning them in order of usage frequency and in order of their hierarchal structural relationships. We show that our work outperforms previously published orders and algorithms. Our algorithm is applicable to any scheduling task where nodes have intrinsic differences in importance and must be visited in topological order.

% Please keep the Author Summary between 150 and 200 words
% Use first person. PLOS ONE authors please skip this step.
% Author Summary not valid for PLOS ONE submissions.
%\section*{Author Summary}
%Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur eget porta erat. Morbi consectetur est vel gravida pretium. Suspendisse ut dui eu ante cursus gravida non sed sem. Nullam sapien tellus, commodo id velit id, eleifend volutpat quam. Phasellus mauris velit, dapibus finibus elementum vel, pulvinar non tellus. Nunc pellentesque pretium diam, quis maximus dolor faucibus id. Nunc convallis sodales ante, ut ullamcorper est egestas vitae. Nam sit amet enim ultrices, ultrices elit pulvinar, volutpat risus.

%\linenumbers

\section*{Introduction}

One of the most fascinating aspects of the Chinese language presents one of the largest barriers to learning it, an irony not lost on generations of students. Chinese characters enchant the learner like little else, but the origins of that enchantment, their elegant, structured complexity and seemingly infinite variety, make learning them a formidable task \cite{ke2001, walker1989}. Mastery is a hard-won thing and is rarely achieved until an advanced stage of study \cite{light1975}.

Becoming functionally literate in Chinese requires memorization of several thousand distinct characters, and the effort involved has profound consequences for the learning process \cite{kane}. An early focus on learning characters can delay the acquisition of productive language skills, while learning them late can inhibit productive learning techniques, such as extensive reading, and also obscure the logic of the language. Either way, the consequences for students are a steep learning curve, high rates of attrition, and a certain preoccupation with methods for learning and remembering characters \cite{marton1}.

The task of learning thousands of distinct symbols is not, however, as difficult as it first appears. There are regularities in the structures of Chinese characters that relate them to their pronunciations and meanings, and also to one another. Around 90\% of characters are semantic-phonetic compounds, in which one part of the character indicates the meaning and the other part the pronunciation. These cues are not always obvious, as meanings and pronunciations have evolved over time, but they remain useful. In a study of 2570 characters taught in Chinese elementary schools, Shu et al. \cite{shu2003} found that 88\% of compound characters had a semantic component that was clearly related to the meaning and 62\% had a phonetic component that provided a useful cue to pronunciation. The level of phonetic regularity is greater than is often appreciated \cite{defrancis, newyorker}.

Compound characters are frequently used as phonetic or semantic components in other characters and so, collectively, Chinese characters form a hierarchal network. At the foundation of this network are primitive symbols, which typically originate from pictographs. Some of these primitives form characters in their own right, while others are used only as components. The structure of the character 照~(zhào, to illuminate) is shown in Fig.~\ref{Fig1}. The decomposition illustrates the typical ways in which phonetic relationships are distorted and how semantic relationships are sometimes rather general or oblique.

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig1.eps}\end{center}
\caption{{\bf Structural decomposition of the character 照.} Primitive characters appear as characters in their own right whereas primitive components do not. The primitive component 灬~is an abbreviated form of the primitive character 火. The parameter $r$ is the SUBTLEX-CH usage frequency rank of the character. Pronunciations are given in pinyin romanization. Note that each character is only assigned a single meaning even though most actually possess a range of broadly related meanings.}
\label{Fig1}
\end{figure}

The semantic-phonetic structure of most Chinese characters makes the learning process somewhat different for native Chinese speakers and second language learners. When Chinese children learn to read and write they already know the spoken language and so the phonetic information can be very useful for making connections between written and spoken forms \cite{marton5}. For second language learners, who are often learning characters at a time when they know little of the language, this information is more difficult to use and the learning process is correspondingly more difficult.

But just as learning characters can be more challenging for second language learners it can also be particularly useful. The Chinese language abounds in homophones, syllables that have identical pronunciations but different meanings (it has many fewer distinct syllables compared to English, around 6 times fewer if one accounts for tones and around 20 times fewer if one neglects them). This gives a potential for ambiguity in the spoken language and acts to obscure some of the logic behind word formation. However, neither of these issues translate into written Chinese because homophones are often represented by different characters. For example, the character 照~of Fig.~\ref{Fig1} is pronounced identically to the unrelated characters 兆~(sign or portent), 罩~(cover) and 棹~(oar). Knowing characters can thus help the learner distinguish between homophones and assign distinct mental identities to the different meanings. This, in turn, can help with understanding and remembering words. For example, the verbs 照应~(zhàoyìng, to coordinate) and 照映~(zhàoyìng, to shine) are pronounced identically, but have differences in meaning that are suggested by the final characters 应~(respond) and 映~(reflect).

There is substantial debate in the literature on how characters should be taught and on the level of knowledge that is required at different educational stages \cite{allen, lam, wang1992, richardson}. This debate, as well as the importance of the problem, is reflected in the wide variety of learning methodologies found in different courses, books and apps. Here we are largely agnostic regarding the best overall approach. Rather, we consider a general question that is relevant to most of them and suggest an answer that is based on broad educational principles. The question we address is the optimum order in which Chinese characters should be learned.

There are two orders that make intuitive sense: in order of usage frequency, from high to low, and in order of network hierarchy, starting with primitives and building up compound characters using components that have already been learned. The first of these follows directly from the goal of the learner but the second merits further discussion. In general terms, the desirability of learning characters in hierarchal order follows from a broad principle of human cognition, that mastery of a complex system rests on mastery of the relevant features of its sub-components \cite{orderbook, chess}. This applies to Chinese characters if one assumes that it is productive to treat them as a complex system rather than as a set of unrelated symbols to be learned by rote. A number of experimental studies indicate that this assumption is valid. They show that orthographic awareness is of critical importance to skilled native readers and in learning to read by both Chinese children and second language learners \cite{shen2004, shen2005, marton4, wang2003, feldman1999, shu1997, connie1999}. These also show that orthographic awareness is present whether or not it is taught explicitly and, among learners, that the extent of the awareness is correlated with performance \cite{shen2007}. We consider learning characters in order of hierarchy to be desirable because we infer that a learning order that explicitly reflects orthographic principles is more likely to generate accurate and productive orthographic awareness in students.

There is, however, necessarily a tension between learning by usage frequency and learning by hierarchy, because frequency is only weakly correlated with character complexity. This behavior can be seen in Fig.~\ref{Fig1}, where, for example, 照~appears around five times more often than its component 刀, and also in Fig.~\ref{Fig2}. Learning characters in order of frequency would therefore often mean learning characters before their components had been learned, whereas learning them in order of hierarchy would often mean learning rarer characters in advance of more common ones.

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig2.eps}\end{center}
\caption{{\bf Usage frequency versus number of unique components for the 1000 most common Chinese characters.} This plot shows the weak relationship between character usage frequency and complexity, the latter represented by the number of unique components used to construct the character. Usage frequency is normalized to 1.0 over the whole usage frequency data set, which encompasses more characters than shown in this plot. The six characters illustrated are the most common in each column. Note that the number of unique components is not the same as visual complexity: the characters 我~and 说~have similar visual complexity (they are composed of similar numbers of strokes) but 我~is conceptually more simple, being, in the OLS character decomposition, composed of two relatively complex primitive components 手~and 戈, compared with the four from which 说~is composed.}
\label{Fig2}
\end{figure}

When devising a learning order one can choose either of the extremes, of frequency or hierarchy, or attempt to find a balance between them in which some common characters are learned in advance of their components. One previous approach that searched for such a balance was a network theory-based approach by Yan et al. \cite{yan}. They demonstrated that an algorithmically-optimized, balanced order can be substantially more efficient than one that follows frequency.

Yan et al. also showed an improvement over pure hierarchal ordering, though somewhat less convincingly. It is unconvincing because they compared their optimized order to only one of many possible hierarchal orders, and there is no reason to believe that the one they choose is representative. Indeed, it will be one of the conclusions of this work that extremely efficient hierarchal orders do exist, ones that can outperform the orders produced by their algorithm.

The tension between frequency and hierarchy is a dominant consideration in determining the learning order but it is not the only one. Small-scale character-to-character patterns are also known to be important, especially for encouraging orthographic awareness \cite{ho2013, xu2014, taftchung}. Patterns can be chosen to emphasize the logic of character construction, by introducing components directly before their compounds, or to emphasize the functional role of components, by presenting their compounds in sets. These patterns are often found in human-curated orders, and especially in books on learning Chinese characters (for example, those of Heisig and Richardson \cite{heisig1, heisig2}). They embody sound educational principles, which can be understood in terms of Marton's variation theory \cite{variations1, variations2, variations3, ho2014}.

Patterns such as these are not present in orders produced by the Yan et al. algorithm. Their procedure generates a degree of character-to-character noise that means that components are rarely adjacent to the compounds that motivate their introduction and sometimes even follow them. This contrasts with the algorithm presented here, which produces orders with a high degree of logical transparency and strong clustering of related characters.

Our algorithm is built on the fundamental assumption that hierarchal orders are the pedagogically desirable way to accumulate usage frequency and we search among this subset of orders for the one that is most efficient. The algorithm is implemented using the conceptual framework of network theory, within which we conceive the network of Chinese characters as a directed analytic graph \cite{li}. The nodes in the graph represent characters and the edges represent the structural relationships between them. We devise a measure of node centrality that relates each character's usage frequency to the effort required to learn it, and order the characters by this measure to provide a first approximation to the optimal learning order. We then sort this list into topological (hierarchal) order using an algorithm designed to minimally disturb the starting order. The algorithm can be applied to any scheduling task where nodes have intrinsic differences in importance and must be visited in topological order.

Following this introduction, we describe the structure of our algorithm in detail, including how we define learning efficiency and how we calculate the cost of learning characters. We discuss the robustness of the algorithm and study the characteristics of the orders it produces.

In the final section we apply our algorithm to a network that is expanded to include Chinese words. Chinese words can be single characters but they are more frequently compounds of two or more. They are the primary units of communication in the Chinese language and so characters, rather like letters in alphabetic scripts, may be considered useful only in so far as they help to build words or act as words themselves. Reflecting this, words can be moved to center stage and, instead of having character usage frequencies drive the acquisition of components, word usage frequencies can be used to drive the acquisition of characters and their components. We explore the results of this more holistic approach.

% You may title this section "Methods" or "Models".
% "Models" is not a valid title for PLoS ONE authors. However, PLoS ONE
% authors may use "Analysis"

\section*{Analysis}

\subsection*{Overview.}

The network of Chinese characters can be represented as a directed analytic graph. Nodes represent characters, with their visual forms, pronunciations and meanings, and edges represent the structural relationships between characters and the nature of those relationships, whether semantic, phonetic or otherwise. Learning Chinese characters means memorizing a productive subset of this network.

Our aim is to derive a character learning order which maximizes learning efficiency. Such an order maximizes cumulative usage frequency while minimizing the effort required to learn it. To this end, we assign a usage frequency to each character along with an estimate for the effort required to learn it, its \textit{learning cost}. Learning costs are calculated using a model that assumes that characters are learned in hierarchal order.

We incorporate usage frequency and learning cost into a measure of character centrality. This measure indicates the relative importance of the character to the learner, prioritizing usage frequency and penalizing learning cost. Ordering characters by this centrality provides a first approximation to the final learning order. This order is approximate because ordering by centrality does not imply ordering by hierarchy, which must be imposed in a separate step.

Hierarchal ordering is imposed using an algorithm designed to topologically sort our centrality-ordered list in a way that minimally disturbs it. Higher-centrality characters are learned first only when allowed topologically.

The algorithm can easily account for characters that are already known to the learner (their learning costs can be set to zero) or characters that are partially known (their learning costs can be suppressed). This capability could be useful in software applications, which could dynamically update the learning order as the student progresses.

The algorithm has potential applications beyond the learning of Chinese characters, and can be applied to any scheduling task where nodes have intrinsic differences in importance yet must be visited in topological order.

\subsection*{Learning efficiency.}

A typical learning scenario is characterized by a fixed available effort, with which the learner seeks to acquire the maximum cumulative usage frequency as rapidly as possible. The learning process can be visualized as a \textit{learning curve} in a space defined by axes of cumulative usage frequency and cumulative learning cost. This is illustrated in Fig.~\ref{Fig3}. Efficient learning curves rise quickly and reach high end-points.

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig3.eps}\end{center}
\caption{{\bf Measures of learning efficiency.} The curves $A$ and $B$ represent two different learning curves. For each curve, the final learning efficiency $\Lambda_f$ is the cumulative usage frequency for a specific cumulative learning cost $C_0$, and the integral learning efficiency $\langle\Lambda\rangle$ is the average cumulative usage frequency between the origin and $C_0$. Curve $A$ has higher $\Lambda_f$ but lower $\langle\Lambda\rangle$. Illustrated values for $\langle\Lambda\rangle$ are approximate.}
\label{Fig3}
\end{figure}

Learning curves for different orders can be compared visually, as in the figure, but it is convenient to parameterize them. We propose a two-parameter scheme.

The first parameter is the cumulative usage frequency at the end of the learning process, once the maximum cumulative learning cost $C_0$ is reached. We call this parameter the \textit{final learning efficiency} $\Lambda_f$. High $\Lambda_f$ is characteristic of efficient learning curves. Note that when comparing curves, it is necessary to use the same usage frequencies for characters that appear in both curves, even though the curves may not cover identical sets of characters. To ensure this, we normalize the entire usage frequency data set to 1.0, which becomes the maximum possible value of $\Lambda_f$.

The second parameter concerns how the maximum cumulative usage frequency is approached. Consider the two curves shown in Fig.~\ref{Fig3}. The curve $A$ has higher $\Lambda_f$ but, over much of the learning process, actually performs less well than curve $B$. Curve $A$ might be one that prioritizes longer-term cumulative frequency at the cost of shorter-term. The difference would be immaterial if the learning process had a short extent in time but this is not typically the case. Learning may take place over many months, during which the learner would likely be exposed to other parts of the language. In this case it might make sense to have a learning curve that rises quickly, even at the cost of some longer-term cumulative usage frequency. We parameterize this using the average cumulative usage frequency, which we call the \textit{integral learning efficiency} $\langle\Lambda\rangle$ and calculate using

\begin{equation}\label{eq:efficiency}
\langle\Lambda\rangle = \frac{1}{C_0} \int_0^{C_0} F(C) \,\mathrm{d}C
\end{equation}
where $F$ is the cumulative usage frequency and $C$ is the cumulative learning cost.

\subsection*{Centrality.}

We define the centrality $\eta_i$ of character $i$ to be

\begin{equation}\label{eq:centrality}
\eta_i = \frac{f_i}{c_i},
\end{equation}
where $f_i$ is the usage frequency and $c_i$ is the learning cost. This quantity is the ratio of the benefit and cost that each character represents to the learner. Learning characters in order of $\eta$ will therefore tend to satisfy the prime concern of the learner, of maximizing cumulative usage frequency and minimizing effort. These learning curves will rise steeply and have high end points, or, in the language of the previous section, be characterized by high integral and final learning efficiencies.

Values for ${f_i}$ can be extracted from corpora of written Chinese. Values for ${c_i}$ are more difficult to assign objectively and we estimate them using a learning model. In our model we use different procedures to assign costs to primitives and compound characters.

The cost $c^{(p)}_i$ of learning a primitive $i$ is taken to be
\begin{equation}\label{eq:alpha}
c^{(p)}_i= 1 + \gamma s_i
\end{equation}
where $s_i$ is the number of strokes that make up the character. Using a $\gamma$ of 0.1 would, for example, mean the cost of learning 口~is 1.3 and the cost of learning 豕~is 1.7. This is a simple approximation to the true learning cost, which would depend on a variety of other things, and likely in complex ways: the learner's familiarity with the strokes that make up the character, their knowledge of similar primitives, and the visual similarity between the primitive and the thing it represents.

The cost $c^{(c)}_i$ of learning a compound $i$ is taken to be
\begin{equation}\label{eq:beta}
c^{(c)}_i= m_i
\end{equation}
where $m_i$ is number of combinations used to build the character. Thus, the cost of learning 的~would be 1, because it is a compound of two components 白~and 勺, and the cost of 茶~would be 2, because it is a compound of three (艹, 人~and 木). For characters that are variants of others (such as 灬~, which is a variant form of 火) we assign a cost of 1. We do not take special account of characters with repeating elements (such as 品) for which we likely overstate the cost.

In this work we take $\gamma= 0.1$. This means that the cost of learning the simplest primitive would be similar to the cost of learning the simplest compound. A typical primitive would be around twice as difficult. Our final learning order depends on the specific value chosen for $\gamma$ but our conclusions are the same for any reasonable value.

Our learning cost model assumes that characters are learned in hierarchal order. When we calculate the cost of learning a compound character we do not include the cost of learning the components themselves, which we assume have already been learned.

Our model implies that the total cost of learning a fixed set of characters is identical for all hierarchal orders. All final learning efficiencies will be identical, with the only differences being in the integral learning efficiencies.

\subsection*{Topological sort.}

Learning characters in order of centrality prioritizes characters that are useful and easy to learn but it does not ensure that characters are ordered according to the character hierarchy. For example, the simple and common 的~will appear a long way in advance of its components 白~and 勺, which appear much less frequently as characters.

We resolve this issue with a sorting algorithm that modifies the centrality-ordered list to ensure that all characters appear before those in which they act as components. The algorithm is illustrated in Fig.~\ref{Fig4} and may be described as follows:

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig4.eps}\end{center}
\caption{{\bf Illustration of the topological sort algorithm.} The ordered list is processed from low to high centrality (right to left in the figure). Once 的~is reached, its components are checked in turn. 白~is found to lie to the right of 的~and so is repositioned to its left. Likewise 勺~is found to the right of 的~and is similarly repositioned. 勺~is positioned to the right of 白~because it has lower centrality. The centralities used in this figure are for illustrative purposes only.}
\label{Fig4}
\end{figure}

\begin{enumerate}
\item Process characters from low to high centrality (right to left in the figure). This is opposite to the order in which characters are learned.
\item Decompose each character into a list of its primitives and all intermediate characters. For example, 照~should be decomposed into 昭, 召, 日, 刀, 口~and 灬.
\item Determine the position of each component in the centrality-ordered list. If the position is to the left of the character then no action is taken. If it is to the right of the character then it should be moved the character's left. Move the character as far left as it will go, while still remaining to the right of all characters with higher centrality.
\end{enumerate}

This procedure ensures that characters are relocated only when necessary and always to the optimum position within the region allowed by the hierarchy. This results in a highly optimized order. However, it is not necessarily the most efficient order possible and we can find special cases, such as the network in Fig.~\ref{Fig5}, where the algorithm does generate a sub-optimal order. We have not found such instances in the real character network but cannot prove that they do not exist.

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig5.eps}\end{center}
\caption{{\bf A network where our algorithm does not return the optimal character order.}  A hypothetical network where the integral learning efficiency of the order generated by the algorithm is lower than another possible order. Letters represent Chinese characters (for example, E is a compound character formed from primitives A and B) and the numbers are centralities. Both orders have identical final learning efficiencies.}
\label{Fig5}
\end{figure}

\subsection*{Source data.}

We use two different representations of the simplified Chinese character network, one compiled with an emphasis on etymological correctness and one with an emphasis on the visual relationships between characters. The former is based on a preliminary version of forthcoming dictionary by \textit{Outlier Linguistic Solutions} \cite{outlier} and the latter is taken from the books \textit{Remembering Simplified Hanzi 1 and 2} by Heisig and Richardson. We refer to these networks as the OLS and HR networks, respectively. The networks have similar coverage: OLS covers 3507 characters and primitives, and HR covers 3250, with 2990 in common between them. The majority of the decompositions are identical and the majority of the differences originate from decisions regarding encoding (a number of components do not have Unicode code points and others can reasonably be represented by more than one code point).

Usage frequency data for characters and words are taken from the SUBTLEX-CH database \cite{subtlex}, which is derived from Chinese film and television subtitles. We chose this database because it is comprehensive and is representative of modern colloquial Chinese. In any practical application of our algorithm, frequency data should be chosen with the specific goals of the learner in mind. The database contains 5938 unique characters and 99121 unique words with frequencies calculated from a total corpus of 46.8 million characters (algorithmically segmented into 33.5 million words). All usage frequencies used in this study are normalized to the whole database. Normalizing in this way, both the OLS and HR networks have cumulative usage frequencies of 0.992.

Stroke counts were taken from the UNIHAN database using the python package cjklib \cite{cjklib}. Where stroke data was unavailable, the number was set to zero.

\section*{Results and discussion}

Fig.~\ref{Fig6} shows the first 85 characters from the learning order derived using the OLS network. The full learning curve for the OLS network is shown in Fig.~\ref{Fig7}, where it is compared to the Yan et al. algorithm and to the fixed character order of Heisig and Richardson. Fig.~\ref{Fig8} shows usage frequencies for the first 85 characters of each of the curves in Fig.~\ref{Fig7}. Learning efficiencies are presented in Table~\ref{Table}.

\begin{figure}%[h]
\begin{center}\includegraphics[width=1.0\textwidth]{Fig6.eps}\end{center}
\caption{{\bf The first 85 characters of our optimized learning order.} Taken together these characters have a cumulative usage frequency of 0.42.}
\label{Fig6}
\end{figure}

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig7.eps}\end{center}
\caption{{\bf Learning curves.} The black and green curves were created using the OLS character decompositions and the two different learning order algorithms. The Yan et al. algorithm was optimized up to a cumulative learning cost of $C_0 = 4000$. The blue curve uses the HR network with Heisig and Richardson's fixed character order. Learning efficiencies are presented in Table~\ref{Table}.}
\label{Fig7}
\end{figure}

\begin{figure}%[h]
\begin{center}\includegraphics[width=1.0\textwidth]{Fig8.eps}\end{center}
\caption{{\bf Usage frequencies for the first 85 characters.} The gray, green and blue bars correspond to the black, green and blue curves in Fig.~\ref{Fig8}. Dark bars represent primitives and light bars represent compounds.}
\label{Fig8}
\end{figure}

The shape of the Heisig and Richardson curve in Fig.~\ref{Fig7} can be understood from the structure of their book. The first half of the curve, between the origin and the large discontinuity, covers their first volume, in which they introduce the bulk of the primitives. These are grouped in chapters according to meaning and each one is followed by all the high-usage compound characters that can be made at that point. This explains the alternating pattern of sharp upward jumps and gentle slopes. The second volume introduces the lower-frequency compounds that are not included in the first. The authors present a hierarchal order which aims for a relatively high $\Lambda_f$ by the end of the first volume but with no particular regard for the shape of the curve. Note that their curve in Fig.~\ref{Fig7} was calculated using SUBTLEX-CH frequency data, which may differ from the frequency data which they used to select their characters and order them.

The curves corresponding to our algorithm and that of Yan et al. were calculated using identical character networks and usage frequencies. We also used identical learning models in order to make a properly normalized comparison. To account for the Yan et al. order being non-hierarchal we extended our learning model to include the cost of any unlearned components, making it similar to the model used in their publication.

We find that our algorithm gives better $\Lambda_f$ and $\langle\Lambda\rangle$. This follows largely from the fact that our order is hierarchal and so there is no inefficiency associated with learning characters by rote and then later re-learning some of the components. But it is also dependent on the particular characteristics of the Chinese character network, because there is no a priori reason why the optimal learning order need be hierarchal. For example, in the extreme, hypothetical case that a small number of complex characters accounted for the vast majority of usage frequency it would be more efficient to ignore the components and just learn them by rote. With such a network the Yan et al. algorithm might perform better because it has access to the non-hierarchal parts of the character order space. Indeed, it remains possible that that a non-hierarchal order is optimal for the character network as it exists.

Our algorithm produces orders that exhibit a high degree of logical transparency. This behavior can be seen in Fig.~\ref{Fig6}, and it follows directly from the algorithm, which tends to cluster components directly before the compounds in which they are used. The sequence 心, 目, 木, 相, 想~is a typical example. Similar sequences appear frequently in the Heisig and Richardson order but rarely in the orders produced by the Yan et al. algorithm.

The Heisig and Richardson order is characterized by the introduction of compound characters in sets that have a particular primitive in common. The pedagogical advantages of this pattern were discussed in the introduction but are not fully realized because of the absence of phonetic information (they do not give character pronunciations) and their deliberate strategy of assigning semantic values to all components, whether or not this is etymologically correct.

Grouping of characters into meaningful sets does not occur in either of the algorithmically-generated orders and its absence offers an avenue for improvement. Nevertheless, the two algorithms do produce orders with markedly different degrees of clustering between related characters (of which grouping of characters into sets might be considered a limiting case). Fig.~\ref{Fig9} shows all three orders compared using two parameters which measure the degree of clustering: the average distance, in number of characters, of each character to its closest preceding component and to the closest character sharing a component. In both these measures the Heisig and Richardson order exhibits the strongest clustering, with ours intermediate between theirs and Yan et al. Along with the logical transparency of our order, we take this to indicate improved pedagogical characteristics compared to Yan et al.

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig9.eps}\end{center}
\caption{{\bf Measures of character clustering.} The top panel shows the average distance, in number of characters, to the closest preceding component. The bottom panel shows the average distance, in number of characters, to another character which shares a component. Curves were generated with a fixed cumulative learning cost of $C_0 = 4000$. Averages below 250 characters are not shown because in this region the averages fluctuate wildly.}
\label{Fig9}
\end{figure}

In summary, we have shown that our algorithm can identify pure hierarchal orders of Chinese characters that give high learning efficiencies. The numerical improvements over Yan et al. are modest, but they are coupled with character-to-character patterns that we suggest are pedagogically advantageous: the order is strictly hierarchal, components are typically introduced immediately before they are used, and there is stronger clustering of related characters.

\begin{table}%[!ht]
%\begin{adjustwidth}{-2.25in}{0in} % Comment out/remove adjustwidth environment if table fits in text column.
\caption{
{\bf Learning curve parameters.} The number of characters learned $N$, final learning efficiency $\Lambda_f$, and integral learning efficiency $\langle\Lambda\rangle$ for reference cumulative learning costs of $C_0 = 500$ and $C_0 = 1500$. The Yan et al. algorithm was optimized up to a cumulative learning cost of $C_0 = 4000$.}
\begin{tabular}{| l | l | l | c | c | c | c | c | c |}
\hline
{\bf Curve}  & {\bf Network} & {\bf Algorithm} & \multicolumn{3}{|c|}{ {\bf $C_0 = 500$} } & \multicolumn{3}{|c|}{ {\bf $C_0 = 1500$} } \\ \hline
  & &  & {\bf $N$}  & {\bf $\Lambda_f$}  & {\bf $\langle\Lambda\rangle$} & {\bf $N$}  & {\bf $\Lambda_f$}  & {\bf $\langle\Lambda\rangle$} \\ \hline
This work    & OLS  &  This work   &  432  &   0.760   &  0.560  &  1333  &  0.938  &  0.770  \\
Yan et al.    & OLS  &  Yan et al.    &  374  &   0.714   &  0.501  &  1199  &  0.923  &  0.740  \\
HR              & HR    &  N/A             &  444  &   0.267   &  0.163  &  1335  &  0.793  &  0.432  \\ \hline
\end{tabular}
\begin{flushleft} %Table notes Phasellus venenatis, tortor nec vestibulum mattis, massa tortor interdum felis, nec pellentesque metus tortor %nec nisl. Ut ornare mauris tellus, vel dapibus arcu suscipit sed.
\end{flushleft}
\label{Table}
%\end{adjustwidth}
\end{table}

\subsection*{Words}

We can expand our analysis to include multiple-character Chinese words by making minor changes to the network and learning model. The network is expanded using multiple-character words from the SUBTLEX-CH word database (limited to the 10000 most common words, for computational convenience). This database is also used for all usage frequencies, including those of characters. Previously, when using character frequencies, the 知~and 道~of 知道~(zh\={\i}dào, to know) had very high frequencies because the compound is common, but when we switch to word frequencies their frequencies become very small compared to the compound - 知道~becomes the important thing to learn and 知~and 道~are only learned in advance because they support it in the hierarchy. The learning model is expanded to account for multiple-character words by assigning them learning costs equal to the number of character combinations required to build them. Thus 知, 道~and 知道~all have learning costs of 1.

The learning order calculated in this way is the learning order for the productive units of communication. Characters and their components are introduced in hierarchal order only when needed to build multiple-character words or when the character forms a single-character word itself. This approach to learning to read and write Chinese has the advantage that the things being learned - the words - can be put to immediate and productive use in reading and writing sentences, activities that helps the learning process. This is not the case for characters, which often only acquire their usage frequencies via expression through words. 知道~is a categorically useful word to know, that can be assigned a clear mental identity and used immediately. Its component characters 知~and 道~are rarely used alone as words and take their character usage frequencies primarily from their presence in this and other, less common compounds. In the character learning order they appear as unrelated characters, ten characters apart, yet 道~can't be used by a learner until 知~has already been learned. In the word learning order they appear together in the logical sequence 首, 道, 矢, 知, 知道.

The word learning curve is shown in Fig. \ref{Fig10}, where it is compared to the curve for characters. The difference between the two is stark, indicating that mastery of characters is substantially easier than mastery of the words they combine to create (something which accords with learner experience).

\begin{figure}%[h]
\begin{center}\includegraphics[width=0.7\textwidth]{Fig10.eps}\end{center}
\caption{{\bf Learning curves for characters and words.} The green curves correspond to HSK word lists for levels 1 to 4 (shorter curve) and 1 to 6 (longer curve). The yellow curves correspond to word lists generated from two levels of beginner readers. All curves were created using the OLS character decompositions.}
\label{Fig10}
\end{figure}

The other curves in the figure show what happens to the word curve when the target set of words is a subset of the wider language. These curves represent realistic situations for the application of our algorithm, in which a student is trying to master the vocabulary required to pass a course or read a particular book. The figure shows curves for the vocabulary lists for levels 1-4 and 1-6 of the HSK Chinese Proficiency Test. This is an exam administered by the Chinese National Office for Teaching Chinese as a Foreign Language (NOTCFL) \cite{hsk} and the lists contain 1200 and 5000 words, respectively. We also show curves corresponding to the lowest two levels of a series of Chinese readers, containing 496 and 977 distinct words, respectively \cite{mc1, mc2}. The text from the readers was segmented into words using the same algorithm used to calculate the word usage frequency list (implemented using PyNLPIR \cite{segment}). These curves necessarily have inferior $\Lambda_f$ and $\langle\Lambda\rangle$ compared with the curve for the wider language but nevertheless represent efficient approaches to the restricted goals.

The algorithm would be similarly useful in managing the transition between different levels of a course or from one book to another. In these situations, the algorithm would provide an efficient way to bridge the gap in vocabulary.

\section*{Supporting Information}

% Include only the SI item label in the paragraph heading. Use the \nameref{label} command to cite SI items in the text.

\paragraph*{S1 File.}
\label{S1_File}
{\bf Input files and character orders.} data.zip is a zipped file containing input data (usage frequencies, decompositions, stroke numbers, target word lists) and final character and word orders. A README within the file contains information on the origin of the input data and guidance on use of the output orders.

\section*{Acknowledgments}

This work was supported by the Shanghai Key Lab for Particle Physics and Cosmology (SKLPPC), Grant No. 15DZ2272100. We acknowledge the generosity of Outlier Linguistic Solutions in sharing part of their dictionary prior to publication.

\nolinenumbers

%\section*{References}
% Either type in your references using
% \begin{thebibliography}{}
% \bibitem{}
% Text
% \end{thebibliography}
%
% OR
%
% Compile your BiBTeX database using our plos2015.bst
% style file and paste the contents of your .bbl file
% here.
%

\begin{thebibliography}{10}

\bibitem{ke2001} Ke C, Wen X, Kotenbeutel C. Report on the 2000 CTLA Articulation Project. Journal of the Chinese Language Teachers Association. 2001; 36(3): 25-60.

\bibitem{walker1989} Walker GLR. Intensive Chinese Curriculum: The EASLI Model. Journal of the Chinese Language Teachers Association. 1989; 24(2): 43-83. %m

\bibitem{light1975} Light T. Controlled Composition and Reading. Journal of the Chinese Language Teachers Association. 1975; 10(2): 70-79. %m

\bibitem{kane} Kane D. The Chinese Language: Its History and Current Usage. Tuttle Publishing; 2006. %phys

\bibitem{marton1} Tse SK, Cheung WM. Chinese and the Learning of Chinese. In: Marton F, Tse SK, Cheung WM, editors. On the Learning of Chinese. Sense Publishers; 2010. pp. 1-8.

\bibitem{shu2003}  Shu H, Chen X, Anderson RC, Wu N, Xuan Y. Properties of School Chinese: Implications for learning to read. Child Development. 2003; 74(1): 27-47.

\bibitem{defrancis} DeFrancis J. Visible Speech: The Diverse Oneness of Writing Systems. University of Hawaii Press; 1989. %phys

\bibitem{newyorker} Chiang T. Bad Character. The New Yorker. 16 May 2016.

\bibitem{marton5} Tse SK, Marton F, Ki WW, Loh EKY. Learning Characters. In: Marton F, Tse SK, Cheung WM, editors. On the Learning of Chinese. Sense Publishers; 2010. pp. 75-102.

%\bibitem{hulstijn} Hulstijin J. Mnemonic Methods in Foreign Language Vocabulary Learning: Theoretical Considerations and Pedagogical Implications. In: Coady J, Huckin T, editors. Second Language Vocabulary Acquisition. Cambridge University Press; 1997. pp 203-224. %phys

\bibitem{wang1992} Wang AY, Thomas MH. The Effect of Imagery-Based Mnemonics on the Long-Term Retention of Chinese Characters. Language Learning. 1992; 42(3): 359-376.

\bibitem{richardson} Richardson TW. Chinese Character Memorization and Literacy: Theoretical and Empirical Perspectives on a Sophisticated Version of an Old Strategy. In: Guder A, Xin J, Yexin W, editors. The Cognition, Learning and Teaching of Chinese Characters. Beijing Language University Press; 2007.

\bibitem{allen} Allen JR. Why Learning to Write Chinese is a Waste of Time: A Modest Proposal. Foreign Language Annals. 2008; 41(2): 237-251.

\bibitem{lam} Lam HC. A Critical Analysis of the Various Ways of Teaching Chinese Characters. Electronic Journal of Foreign Language Teaching. 2011; 8(1): 57-70.

\bibitem{orderbook} Ritter FE, Nerb J, Lehtinen E, O'Shea T, editors. In Order to Learn: How the Sequence of Topics Influences Learning. Oxford University Press; 2007. %phys

\bibitem{chess} Chase WG, Simon HA. Perception in Chess. Cognitive Psychology. 1973; 4: 55-81.

%\bibitem{ellis2003} Ellis NC. Constructions, Chunking, and Connectionism: The Emergence of Second Language Structure. In Doughty CJ, Long MH, editors. The Handbook of Second Language Acquisition. Oxford: Blackwell; 2001.

\bibitem{shen2004} Shen HH. Level of Cognitive Processing: Effects on Character Learning Among Non-Native Learners of Chinese as a Foreign Language. Language and Education. 2004; 18(2): 167-182.

\bibitem{shen2005} Shen HH. An Investigation of Chinese-Character Learning Strategies Among Non-Native Speakers of Chinese. System. 2005; 33: 49-68.

\bibitem{wang2003} Wang M, Perfetti CA, Liu Y. Alphabetic Readers Quickly Acquire Orthographic Structure in Learning to Read Chinese. Scientific Studies of Learning. 2003; 7(2): 183-208.

\bibitem{feldman1999} Feldman LB, Siok WWT. Semantic Radicals Contribute to the Visual Identification of Chinese Characters. Journal of Memory and Language. 1999; 40: 559-576.

\bibitem{shu1997} Hua S, Anderson RC. Role of Radical Awareness in the Character and Word Acquisition of Chinese Children. Reading Research Quarterly. 1997; 32(1): 78-89.

\bibitem{marton4} Lam HC. Orthographic Awareness. In: Marton F, Tse SK, Cheung WM, editors. On the Learning of Chinese. Sense Publishers; 2010. pp. 53-73.

\bibitem{connie1999} Ho CSH, Wong WL, Chan WS. The Use of Orthographic Analogies in Learning to Read Chinese. Journal of Child Psychology and Psychiatry. 1999; 40(3): 393-403.

\bibitem{shen2007} Shen HH, Ke C. Radical Awareness and Word Acquisition Among Nonnative Learners of Chinese. The Modern Language Journal. 2007; 91(1): 97-111.

\bibitem{yan}
Yan X, Fan Y, Di Z, Havlin S, Wu J. Efficient Learning Strategy of Chinese Characters Based on Network Approach. PLoS ONE. 2013; 8: e69745. doi:10.1371/journal.pone.0069745

\bibitem{xu2014} Xu Y, Perfetti CA, Chang LY. The Effect of Radical-Based Grouping in Character Learning in Chinese as a Foreign Language. The Modern Language Journal. 2014; 98(3): 773-793.

\bibitem{taftchung} Taft M, Chung K. Using Radicals in Teaching Chinese Characters to Second Language Learners. Psychologia. 1999; 42: 243-251.

\bibitem{ho2013} Ho CL, Bik MAT. Drawing on the Variation Theory to Enhance Students' Learning of Chinese Characters. Instructional Science. 2013; 41: 955-974.

\bibitem{heisig1}
Heisig JW, Richardson TW. Remembering Simplified Hanzi: Book 1, How Not to Forget the Meaning and Writing of Chinese Characters. University of Hawaii Press; 2008.

\bibitem{heisig2}
Heisig JW, Richardson TW. Remembering Simplified Hanzi: Book 2, How Not to Forget the Meaning and Writing of Chinese Characters. University of Hawaii Press; 2012.

\bibitem{variations1} Marton F, Booth S. Learning and Awareness. Lawrence Eribaum Associates; 1997.

\bibitem{variations2} Marton F, Pang MF. On Some Necessary Conditions of Learning. The Journal of the
Learning Sciences. 2006; 15(2): 193-220.

\bibitem{variations3} Ling LM, Marton F. Towards a Science of the Art of Teaching: Using Variation Theory as a Guiding Principle of Pedagogical Design. International Journal for Lesson and Learning Studies. 2012; 1(1): 7-22.

\bibitem{ho2014} Ho CL. Elaborating the Concepts of Part and Whole in Variation Theory: The Case of Learning Chinese Characters. Scandinavian Journal of Educational Research. 2014; 58(3): 337-360.

\bibitem{li}
Li J, Zhou J. Chinese Character Structure Analysis Based on Complex Networks. Physica A. 2007; 380.

\bibitem{outlier} Henson A, Renfroe J. Outlier Dictionary. Outlier Linguistics Solutions USA, LLC; 2015.

\bibitem{subtlex} Cai Q, Brysbaert M. SUBTLEX-CH: Chinese Word and Character Frequencies Based on Film Subtitles. PLoS ONE. 2010; 5: e10729. doi: 10.1371/journal.pone.0010729

\bibitem{cjklib} cjklib 0.3.2 Available: https://pypi.python.org/pypi/cjklib. Accessed: 1 May 2016.

\bibitem{hsk} HSK Chinese Proficiency Test. Available: http://www.chinesetest.cn. Accessed: 16 January 2016.

\bibitem{mc1} Burnett FH. The Secret Garden: Mandarin Companion Graded Readers Level 1. Mind Spark Press LLC; 2015.

\bibitem{mc2} Dickens C. Great Expectations (Parts 1 and 2): Mandarin Companion Graded Readers Level 2. Mind Spark Press LLC; 2016.

\bibitem{segment} PyNLPIR 0.4.4 Available: https://pypi.python.org/pypi/PyNLPIR. Accessed: 1 May 2016.

\end{thebibliography}

% ADDITION Chinese support
\end{CJK*}
```


abstract: Deep learning models, such as the fully convolutional network (FCN), have been widely used in 3D biomedical segmentation and achieved state-of-the-art performance. Multiple modalities are often used for disease diagnosis and quantification. Two approaches are widely used in the literature to fuse multiple modalities in the segmentation networks: early-fusion (which stacks multiple modalities as different input channels) and late-fusion (which fuses the segmentation results from different modalities at the very end). These fusion methods easily suffer from the cross-modal interference caused by the input modalities which have wide variations. To address the problem, we propose a novel deep learning architecture, namely OctopusNet, to better leverage and fuse the information contained in multi-modalities. The proposed framework employs a separate encoder for each modality for feature extraction and exploits a hyper-fusion decoder to fuse the extracted features while avoiding feature explosion. We evaluate the proposed OctopusNet on two publicly available datasets, i.e. ISLES-2018 and MRBrainS-2013. The experimental results show that our framework outperforms the commonly-used feature fusion approaches and yields the state-of-the-art segmentation accuracy.
author: Yu Chen[^1]; Jiawei Chen; Dong Wei; Yuexiang Li; Yefeng Zheng
title: OctopusNet: A Deep Learning Segmentation Network for Multi-modal Medical Images

# Introduction

Recent years have witnessed the rapid development of deep learning technique. Deep learning models have been widely used for medical image segmentation and achieved impressive performance . Compared with natural images, medical images, e.g. computed tomography (CT) and magnetic resonance imaging (MRI), often have a lot of scanning protocols in its toolbox and each protocol may reveal a different property (often complementary to other protocols) of the underlying tissue. For examples, to assess ischemic stroke lesion, three modalities using perfusion imaging are commonly captured, i.e. cerebral blood volume (CBV), cerebral blood flow (CBF), and time to peak of the residue function (Tmax). Those modal images may contain different clinical interpretation.

To exploit the multi-modal medical data, two fusion approaches are widely-used by current deep learning networks, i.e. stacking multiple modalities as different input channels (early-fusion, Fig. (a)) and fusing the outputs of networks from different modalities (late-fusion, Fig. (b)) . Neither fusion approach is optimal in using the complementary information from multiple input modalities. Take the perfusion CT for ischemic stroke lesion segmentation as an example. As shown in Fig. (c), four modalities, i.e. CBV, CBF and MTT, and Tmax, are captured. It can be observed from the four modalities that the lesion area in CBV and CBF is darker compared to the normal area, while it is lighter in the modalities of MTT and Tmax. Consequently, the information in different modalities may be wrongly fused if we simply adopt the early-fusion approach. On the other hand, although the late fusion approach adopts the separate encoder-decoder for each modality, the whole network is computational expensive and difficult to converge.

In this paper, we propose a novel segmentation network, namely OctopusNet, which effectively leverages the information contained in multi-modal medical images. Instead of fusing multi-modal images at the input stage, we exploit an individual encoder for each modality and fuse the feature maps generated by middle stages of the network, which specifically extracts features from each modality and explicitly considers the correlations between different modalities. As the modalities are separately encoded, the proposed OctopusNet adopts a novel feature fusion module, namely a hyper-fusion decoder, to merge the feature maps and avoid feature explosion. Extensive comparison experiments are conducted on multi-modal datasets. The results demonstrate the outstanding segmentation performance of the proposed OctopusNet.

# OctopusNet

In this section, we introduce the detailed information of our OctopusNet. The framework of our OctopusNet is shown in Fig. (c). The colored cubes refer to the feature maps generated at different stages of the framework. Our OctopusNet addresses the problem of cross-modal interference, occured in early-fusion, by extracting features from the modalities using separate modal encoders, which can be any CNN architecture, e.g. VGG , ResNet or DenseNet . As shown in Fig. (c), the feature maps generated at different stages of modal encoders are concatenated and fed to the hyper-fusion decoder. The proposed decoder uses hyper-fusion modules to fuse the feature maps from different modalities and avoid the problem of feature explosion. Compared to the late-fusion approach with four separate decoders, the hyper-fusion decoder more effectively fuses cross-modal information and reduces the computational cost. The decoder upsamples the high-level low-resolution feature maps back to the original resolution in the same way as , and yields the segmentation result.

## Modal encoder

As aforementioned, the modal encoder can be chosen from any widely-used network architectures, e.g. DenseNet. In our experiments, DenseNet-161 usually yields better segmentation accuracy compared to that of VGG and ResNet. Therefore, we take DenseNet-161 as an example to illustrate the pipeline of extracting feature maps from different modalities. The detailed information of network architecture of DenseNet-161 can be found in . The colored cubes in each modal encoder in Fig. (c) are the feature maps generated by different stages of DenseNet-161, which correspond to the ones from Dense Block (1) - (4). To better leverage the explicit information contained in different modalities, all the extracted feature maps are concatenated together and fed to the hyper-fusion decoder for feature fusion and distilling.

## Hyper-fusion decoder

We propose a novel hyper-fusion decoder to decode and upsample the high-level low-resolution feature maps back to the original resolution of input and yield the segmentation result. As shown in Fig. (c), the decoder adopts a *hyper-fusion module* for feature distilling, *deep supervision* for better training convergence, and concatenates the fused feature maps (purple cubes) to the upsampled ones (yellow cubes) to produce segmentation result.

As the network goes deeper, the number of feature maps increases. Consequently, the concatenation of multi-modal feature maps easily causes a problem of feature explosion. For example, assuming there are N modalities as input, the number of concatenated feature maps generated by Dense Block (4) of modal encoders is $2208 \times N$, which requires high cost of computation and memory consumption. The hyper-fusion module (1) - (4) in Fig. (c) is a $1 \times 1$ convolution, which has the same number of channels to that of feature maps from Dense Block (1) - (4). Therefore, the N concatenated feature maps can be accordingly fused and compacted to single ones using hyper-fusion modules (the purple cubes in Fig. (c)).

The proposed OctopusNet is an end-to-end framework, which means the multiple modal encoders and hyper-fusion decoder are simultaneously trained and updated. However, the networks adopted for modal encoders are usually extremely deep, resulting in a difficulty for training convergence only using a single supervision signal at the very end of a long pipeline. Hence, we added a weak supervision signal to the deepest node of OctopusNet, i.e. the elongated purple cube at the bottom. Assuming DenseNet-161 is adopted as the modal encoder, the size of bottom purple cube is $2208 \times 7 \times 7$. In this situation, a $1 \times 1$ convolution is used to transform the cube to $1 \times 7 \times 7$ and the original supervision signal is resized from $224 \times 224$ to $7 \times 7$ to be as weak supervision.

# Experiments

We evaluate the performance of the proposed OctopusNet on publicly available datasets from two challenges, namely ISLES-2018[^2] and MRBrainS-2013[^3], and compare with the early-fusion and late-fusion approaches. Though our result is competitive to the top-performance of the ISLES-2018 challenge, the purpose of the experiments is not to win the challenges. Our main purpose is to demonstrate the effectiveness of the proposed fusion approach and compare with other widely used alternative strategies. Our approach is complementary to, and can be easily integrated into, other FCN based multi-modal segmentation approaches.

## Datasets

Ischemic stroke lesions segmentation (ISLES) is a competition consecutively held since 2015 . In ISLES-2018, the challenge organizer released a new dataset, which is composed of six modalities, including diffusion weighted imaging (DWI) MRI, computed tomography (CT) and four perfusion scans, i.e. mean transit time (MTT), time to peak of the residue function (Tmax), cerebral blood flow (CBF) and cerebral blood volume (CBV). The ISLES-2018 competition provides 94 sets of multi-modal data for training and 62 sets for test. As the ground truth of the test set is not available, participants need to submit their prediction to the online system for performance evaluation.

The dataset has two main challenging issues. First, the appearances of lesion areas among different modalities are widely varied. The lesion area in MTT and Tmax is brighter than the normal area, while it is dark in the modalities of CBF and CBV, as shown in Fig. (c). Second, the ISLES-2018 data does not have a uniform size. Though each slice has a fixed size of $256 \times 256$ pixels, the number of slices contained in a volume varies from 2 to 22. Most ISLES-2018 volumes only have two slices, which presents a difficulty to adapt a 3D segmentation framework to the dataset.

The MRBrainS-2013 dataset contains five sets of multi-modal brain images, in which the brain tissues, i.e. gray matter, white matter and cerebrospinal fluid, are fully annotated. Three registered modalities, i.e. T1-weighted scan (T1), T1-weighted inversion recovery scan (T1_IR) and T2-weighted fluid attenuated inversion recovery scan (T2_FLAIR) are provided. The volumes of the dataset are in an uniform size of $240 \times 240 \times 48$ voxels.

The proposed OctopusNet may have different architectures regarding to the input data. As most ISLES-2018 data has a couple of slices, we develop a 2.5D OctopusNet instead of 3D. Three consecutive slices from a volume are extracted and fed to 2D modal encoders as inputs. The first and last slices of the volume are duplicated for padding. In this setting, modal encoders can be pretrained on the ImageNet dataset for better training convergence. Our OctopusNet is implemented using PyTorch. The initial learning rate is set to 0.7 and divided by 10 after every 35 epochs. The network is optimized by stochastic gradient descent (SGD). The used datasets have different number of modalities. Consequently, the proposed OctopusNet involves different numbers of modal encoders for the ISLES-2018 and MRBrainS-2013.

Take the ISLES-2018 as an example. As the ISLES-2018 test set does not provide the DWI modality, five original modalities, i.e. CT, MTT, Tmax, CBF and CBV, are adopted as input for OctopusNet. Furthermore, the lesion area is not clearly visible in the modalities of CT, CBF and CBV. Hence, these three modalities are enhanced by histogram equalization. Finally, five original modalities and three enhanced modalities are fed to the OctopusNet[^4]. The input size of each modal encoder is $256 \times 256 \times 3$.

## Performance Analysis

We perform a five-fold cross validation on the ISLES-2018 and MRBrainS-2013 training set to evaluate the performance of our OctopusNet. All the experiments are repeated three times to reduce the influence caused by random nature of network training. Hence, the results reported in the paper are the the average results of three repeated experiments. For the convenience of comparison, the frameworks using baseline fusion approaches (early- and late-fusion) in our experiments are in the same setting to that of OctopusNet, e.g. the input format of different modalities. Henceforth, the fusion approach adopted in our OctopusNet is named as Octopus-fusion. The Dice coefficient, which measures the spatial overlap index between the segmentation results and ground truths, is adopted as the metric to evlauate the segmentation accuracy.

|                                       |  VGG-16   | ResNet-50  | DenseNet-161  |
|:--------------------------------------|:---------:|:----------:|:-------------:|
| **Single modality (Tmax)**            |   44.97   |   44.03    |     45.83     |
| **Early-fusion**                      |   53.38   |   53.99    |     53.82     |
| **Late-fusion**                       |   53.73   |   55.39    |     53.86     |
| **Octopus-fusion**                    | **55.71** | **57.33**  |   **57.72**   |
| **Octopus-fusion + deep supervision** |    \-     |     \-     |     57.90     |

Dice coefficient (%) of lesion areas of ISLES-2018 (average of five-fold cross validation).

As aforementioned, the modal encoder can be chosen from widely used deep learning networks. To evaluate the generalization capability of Octopus-fusion, several network architectures, e.g. VGG-16 , ResNet-50 and Dense-Net-161 , are adopted as the modal encoder and trained with different fusion approaches on the ISLES-2018 dataset. The results are listed in Table . To evaluate the improvement produced by the usage of multi-modal images, we also report segmentation accuracy using a single modality. Due to the space limit, Table  only lists the result of the best single modality (i.e., Tmax). Due to the lack of information contained in extra modalities, the frameworks using single modality only yield Dice coefficients around 44%, which are about 9% lower than that of multi-modal frameworks. For the early-fusion approach, Table  shows that the accuracies of all three backbone networks are quite similar with the deep networks (i.e., DenseNet-161 and ResNet-50) slightly outperforming the shallow network of VGG-16.

For the late-fusion approach, as it involves multiple encoder-decoder architectures for different modalities, the explicit information contained in multi-modal data can be better extracted. Hence, accuracy of the late-fusion approach surpasses that of early-fusion with the same modal encoders. However, DenseNet-161 only gains marginal improvement, i.e. 0.04%, by switching from early-fusion to late-fusion. The reason for that is the network depth of DenseNet-161 is extremely deep, which makes it difficult to simultaneously well train multiple fully convolutional DenseNet-161 branches in the late-fusion approach. Compared to the results of early- and late-fusion, our Octopus-fusion approach significantly boosts the accuracy of modal encoders. The best lesion segmentation result is achieved by the Octopus-fusion DenseNet-161, i.e. a Dice of 57.72%. By adding the *deep supervision* signal, the segmentation accuracy is further increased to 57.90%, which is 2.51% higher than that of the best-performance among benchmarking algorithms (late-fusion with ResNet-50).

**ISLES-2018 Challenge.** We participated the ISLES-2018 competition. The proposed OctopusNet using DenseNet-161 achieved an average Dice of 48%, which ranked the third-place of ISLES-2018 challenge[^5]. We notice that, for all participating teams, there is a gap between validation and test accuracy. One possible reason is that the test set contains more small lesions, where are difficult to segment accurately for all algorithms. Additionally, the top approaches reported that they used extra modalities, e.g. 4D perfusion CT (ranked 1$^{st}$ with Dice of 51%) and synthesized DWI (ranked 2$^{nd}$ with Dice of 49%), which were not adopted by our OctopusNet.

We also conduct experiments on MRBrainS-2013 to compare the performances of different fusion approaches for the task of brain tissue segmentation. The three original modalities of MRBrainS-2013 are directly employed as input for the proposed OctopusNet. The best-performer on ISLES-2018, i.e. DenseNet-161, is adopted as the backbone of modal encoder. The input size of each modal encoder is $240 \times 240 \times 3$. The Dice coefficients for different tissues, including CSF, gray matter and white matter, produced by different fusion approaches are listed in Table . The average Dice (Ave. Dice) is calculated by averaging the Dice coefficients of three tissues.

|                                |    CSF    | Gray matter | White matter | Ave. Dice |
|:-------------------------------|:---------:|:-----------:|:------------:|:---------:|
| **Single modality (T1)**       |   78.46   |    81.37    |    85.69     |   81.84   |
| **Single modality (T1_IR)**    |   75.95   |    77.51    |    81.92     |   78.46   |
| **Single modality (T2_FLAIR)** |   74.00   |    75.63    |    77.32     |   75.65   |
| **Early-fusion**               |   79.05   |    80.58    |    83.56     |   81.07   |
| **Late-fusion**                |   79.16   |    81.41    |    84.71     |   81.76   |
| **Octopus-fusion**             | **80.59** |  **82.12**  |  **86.05**   | **82.92** |

Dice coefficients (%) yielded by different fusion approaches for each brain tissue of MRBrainS-2013 (average of five-fold cross validation).

The framework using single modality is also evaluated for comparison. It is interesting to see that, for the gray matter and white matter, the best single modality (T1) produces even higher segmentation accuracy than early-fusion. The reason for that may be the physicians mark the annotation of the gray and white matter primarily using the T1 scans, while the T1_IR and T2_FLAIR scans usually provide additional information for the annotation of outer border of CSF and white matter lesion, respectively. Therefore, most information contained in the extra modalities, i.e. T1_IR and T2_FLAIR, may be seen as noises for the brain tissue segmentation. The late-fusion approach yields similar average segmentation accuracy to using T1 only (81.76% vs. 81.84%). The reason for that may be the post-fusion approach performs information fusion too late; Therefore, it can not fully utilize the complementary information among multiple modalities. Oppositely, by using the proposed Octopus-fusion, the average segmentation accuracy increases to 82.92% and improvement is observed for all tissues, which illustrates that our Octopus-fusion can effectively extract useful information from each modality and prevent the cross-modal interference caused by irrelevant information. An additional observation is that CSF consistently benefits from multi-modality fusion using any fusion strategy, which is concordant to the annotation process of physicians. Again, Octopus-fusion achieves the largest boost in segmentation accuracy of CSF, i.e., $+2.13\%$.

# Conclusion

In this paper, we presented a novel deep learning network architecture, namely OctopusNet, for multi-modal medical image segmentation. The proposed OctopusNet adopted a separate modal encoder for each modality to explicitly extract features and a hyper-fusion decoder to fuse the features, avoiding the problem of feature explosion. The proposed OctopusNet was evaluated on two publicly available datasets. The experimental results demonstrated that our OctopusNet was a general network architecture, which can provide excellent performance for various segmentation tasks of multi-modal medical data.

[^1]: This work was done when Yu Chen was an intern at YouTu Lab

[^2]: http://www.isles-challenge.org/

[^3]: http://mrbrains13.isi.uu.nl/index.php

[^4]: This network has an octopus shape with a body (the decoder) and eight arms (the encoders). This is where the name, OctopusNet, comes from.

[^5]: https://www.smir.ch/ISLES/Start2018