abstract: This paper models unsupervised learning of an identity-based pattern (or copying) in speech called reduplication from raw continuous data with deep convolutional neural networks. We use the ciwGAN architecture in which learning of meaningful representations in speech emerges from a requirement that the CNNs generate informative data. We propose a technique to wug-test CNNs trained on speech and, based on four generative tests, argue that the network learns to represent an identity-based pattern in its latent space. By manipulating only two categorical variables in the latent space, we can actively turn an unreduplicated form into a reduplicated form with no other substantial changes to the output in the majority of cases. We also argue that the network extends the identity-based pattern to unobserved data. Exploration of how meaningful representations of identity-based patterns emerge in CNNs and how the latent space variables outside of the training range correlate with identity-based patterns in the output has general implications for neural network interpretability.
author: Gašper Beguš
University of California, Berkeley
email@example.com
bibliography: begusGANbib.bib
title: Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication

#  Introduction

The relationship between symbolic representations and connectionism has been subject to ongoing discussions in computational cognitive science. Phonology offers a unique testing ground in this debate as it is concerned with the first discretization that human language users perform: from continuous phonetic data to discretized mental representations of meaning-distinguishing sounds called *phonemes*.

Identity-based patterns, repetition, or copying have been at the center of this debate . Reduplication is a morphophonological process where phonological content (phonemes) get copied from a word (also called the base) with some added meaning . It can be total, which means that all phonemes in a word get copied (e.g. /pula/ $\rightarrow$ \[**pula**-pula\]), or partial, where only a subset of segments gets copied (e.g. // $\rightarrow$ \[**pu**-pula\]).

Reduplication is indeed unique among processes in natural language because combining learned entities based on training data distributions does not yield the desired outputs. For example, a learner can be presented with pairs of bare and reduplicated words, such as /pala/ $\sim$ /papala/ and /tala/ $\sim$ /tatala/. The learner can then be tested on providing a reduplicated variant of a novel unobserved item with an initial sound /k/ that they have not been exposed to (e.g. /kala/). If the learner learns the reduplication pattern, they will output \[kakala\]. If the learner simply learns that /pa/ and /ta/ are optional constituents that can be attached to words based on data distribution, they will output \[pakala\] or \[takala\]. Reduplication is thus an identity-based pattern (similar to copying), which is computationally more challenging to learn , both in connectionist and non-connectionist frameworks . In /k$_i$a$_j$k$_i$a$_j$la/, the two sounds in the reduplicative morpheme, /k$_i$/ and /a$_j$/, need to be in an identity relationship with the first two segments of the base, /k$_i$/ and /a$_j$/, and the learner needs to copy rather than recombine learned elements.

argue that connectionist models such as simple neural networks are unable to learn a simple reduplication pattern that 7-month old human infants are able to learn (see also ). According to , the behavioral outcomes of their experiments cannot be modeled by simple counting, attention to statistical trends in the input, attention to repetition, or connectionist (simple neural network) computational models. Instead, they argue, the results support the claim that human infants employ "algebraic rules" to learn reduplication patterns (for a discussion, see, among others, ).

With the development of neural network architectures, several studies revisited the claim that neural networks are unable to learn reduplicative patterns , arguing that identity-based patterns can indeed be learned with more complex architectures.[^1] All these computational experiments, however, operate on an already discretized level and most of these experiments model reduplication with supervised learning.

Examples like \[**pu**-pula\] and \[**pula**-pula\] are discretized representations of reduplication, using characters to represent continuous sounds. Most, if not all, computational models of reduplication, to the author's knowledge, model reduplication as character or feature manipulation (the inputs to the models are either characters representing phones or phonemes or discrete binary featural representations of phonemes). For example, a seq2seq model treats reduplication as a pairing between the input unreduplicated sequence of "characters" (such as /tala/) and an output — a reduplicated sequence (such as /tatala/). Already abstracted and discretized phonemes or "characters", however, are not the primary input to language-learning infants. The primary linguistic data of most hearing infants is raw continuous speech. Hearing infant learners need to acquire reduplication from continuous speech data that is substantially more complex than already discretized characters or binary features.

Furthermore, most of the existing models of reduplication are also supervised. Seq2seq models, for example, are fully supervised: the training consists of pairs of unreduplicated (input) and reduplicated strings of characters or binary features (output). While the performance can be tested on unobserved data or even on unobserved segments, the training is nevertheless supervised. Human language learners do not have access to input-output pairings: they are only presented with positive, surface, and continuous acoustic data. While equivalents of copying/identity-based patterns can be constructed in the visual domain, we are not aware of studies that test identity-based visual patterns with deep convolutional neural networks.

In this paper, we model reduplication, one of the computationally most challenging processes, from raw unlabeled acoustic data with deep convolutional networks in the GAN framework. The advantage of the GAN framework for cognitive modeling is that the network has to learn to output raw acoustic data from a latent noise distribution without directly accessing the training data. We argue that CNNs discretize continuous phonetic data and encode linguistically meaningful units into individual latent variables. The emergence of a discretized representation of an identity-based pattern (reduplication) is induced by a model which forces the Generator network to output informative data (ciwGAN; Section ). Additionally, we add inductive bias towards symbolic-like representations by binarizing code variables with which the Generator encodes meaningful representations. We also test whether a deep convolutional network learns reduplication without the two inductive biases (without the requirement on the Generator to output informative data and without binarization of the latent space) in the bare WaveGAN architecture (Section ).

The experiments bear implications for the discussion between symbolic and connectionist approaches to language modelling by testing the emergence of rule-like symbolic representations within the connectionist framework from raw speech in an unsupervised manner. Results of the experiments suggest that both models, ciwGAN and WaveGAN learn the identity-based patterns, but inductive biases for informative representation and binarization facilitate learning and yield better results. We discuss properties of symbolic-like representations and how they emerge in the models: discretization, causality (in the sense that manipulation of individual elements results in desired outcome), and categoricity (for discussions on these and other aspects of the debate, see ).

How can we test learning of reduplication in a deep convolutional network that is trained only on raw positive data? We propose a technique to test the ability of the Generator to produce forms absent from the training data set. For example, we train the networks on acoustic data of items such as /pala/ $\sim$ /papala/ and /tala/ $\sim$ /tatala/, but test reduplication on acoustic forms of items such as /sala/, which is never reduplicated in the training data. Using the technique proposed in , we can identify latent variables that correspond to some phonetic or phonological representation such as reduplication. By manipulating and interpolating a single latent variable, we can actively generate data with and without reduplication. In fact, we can observe a direct relationship between a single latent variable (out of 100) and reduplication that with interpolation gradually disappears from the output. Additionally, we can identify latent variables that correspond to \[s\] in the output. By forcing both reduplication and \[s\] in the output through latent space manipulation, we can "wug-test" the network's learning of reduplication on unobserved data. In other words, we can observe what the network will output if we force it to output reduplication and an \[s\] at the same time. A comparison of generated outputs with human outputs that were withheld from training reveals a high degree of similarity. We perform an additional computational experiment to replicate the results from the first experiment (from Section ). In the replication experiment, evidence for learning of the reduplicative pattern also emerges. To the author's knowledge, this is the first attempt to model reduplication with neural network architectures trained on raw acoustic speech data.

The computational experiments reveal another property about representation learning in deep neural networks: we argue that the network extracts information in the training data and represents a continuous acoustic identity-based pattern with discretized representation. Out of 100 variables, the network encodes reduplication with one or two variables, which is suggested by the fact that a small subset of variables are substantially more strongly correlated with presence of reduplication. In other words, there is a near categorical drop in regression estimates between one variable and the rest of the latent space. Setting the identified variables to values well beyond the training range results in near categorical presence of a desired variable in the output. This technique (proposed for non-identity-based patterns in ) allows us to directly explore how the networks encode dependencies in data, their underlying values, and interactions between variables, and thus get a better understanding of how exactly deep convolutional networks encode meaningful representations.

Recent developments in zero-resource speech modeling enable modeling of speech processes in an unsupervised manner from raw acoustic data. Several proposals exist for modeling unsupervised lexical learning that include generative models such as variational autoencoders and GANs . This framework allows not only unsupervised lexical term discovery, but also phone-level identification . While zero-resource speech modeling has yielded promising results in unsupervised labeling, the proposals generally do not model phonological or morphophonological processes. This paper thus also tests applicability of the unsupervised speech processing framework for cognitive modeling and network interpretability.

#  Model

Generative Adversarial Networks are a neural network architecture with two main components: the Generator network and the Discriminator network. The Generator is trained on generating data from some latent space that is randomly distributed. The Discriminator takes real training data and the Generator's outputs and estimates which inputs are real and which are generated. The minimax training, where the Generator is trained on maximizing the Discriminator's error rate and the Discriminator is trained on minimizing its own error rate, results in the Generator network outputting data such that the Discriminator's success in distinguishing them from real data is low. It has been shown that GANs not only learn to produce innovative data that resemble speech, but also learn to encode phonetic and phonological representations in the latent space . The major advantage of the GAN architecture for modeling speech is that the Generator network does not have direct access to the training data and is not trained on replicating data (unlike in the autoencoder architecture; ). Instead, the network has to learn to generate data from noise in a completely unsupervised manner — without ever directly accessing the training data.

In the first experiment, we use the ciwGAN (Categorical InfoWaveGAN) model proposed in . The ciwGAN model combines the WaveGAN and InfoGAN architectures. WaveGAN, proposed by , is a Deep Convolutional Generative Adversarial Network (DCGAN; proposed by ) adapted for time-series audio data. The basic architecture is the same as in DCGAN, the main difference being that in the WaveGAN proposal, the deep convolutional networks take one-dimensional time-series data as inputs or outputs. The structure of the Generator and the Discriminator networks in the ciwGAN architecture are taken from . InfoGAN is an extension of the GAN architecture that aims to maximize mutual information between the latent space and generated outputs. The Discriminator/Q-network learns to retrieve the Generator's latent categorical or continuous codes in addition to estimating realness of generated outputs and real training data.

proposes a model that combines these two proposals and introduces a new latent space structure (in the fiwGAN architecture). Because we are primarily interested in simple binary classification between bare and reduplicated forms, we use the ciwGAN variant of the proposal. The model introduces a separate deep convolutional Q-network that learns to retrieve the Generator's internal representations. Separating the Discriminator and the Q-network into two networks is advantageous from the cognitive modeling perspective: the architecture features a separate network that models speech production (the Generator) and a separate network that models speech categorization (the Q-network). The latter introduces an inductive bias that forces the Generator to output informative data and encode linguistically meaningful properties into its code variables. The network learns to generate data such that by manipulating these code variables, we can force the desired linguistic property in the output .

The architecture involves three networks: the Generator that takes latent codes (a one-hot vector) and uniformly distributed $z$-variables and generates waveforms, a Discriminator that distinguishes real from generated outputs, and a Q-network that takes generated outputs and estimates the latent code (one-hot vector) used by the Generator. More specifically, the Generator network is a deep convolutional network that takes as its input 100 latent variables (see Figure ).[^2] Two of the 100 variables are code variables ($c_1$ and $c_2$) that constitute a one-hot vector. The remaining 98 $z$-variables are uniformly distributed on the interval $(-1, 1)$. The Generator learns to take as the input the 2 code variables and the 98 latent variables and output 16384 samples that constitute just over one second of audio file sampled at 16 kHz through five convolutional layers. The Discriminator network takes real and generated data (again in the form of 16384 samples that constitute just over one second of audio file) and learns to estimate the Wasserstein distance between generated and real data (according to the proposal in ) through five convolutional layers. In the majority of InfoGAN proposals, the Discriminator and the Q-network share convolutions. introduces a separate Q-network (also in ).[^3]

The Q-network is in its structure identical to the Discriminator network, but the final layer is fully connected to nodes that correspond to the number of categorical variables . In the ciwGAN architecture, the Q-network is trained on estimating the latent code variables with a softmax function . In other words, the Q-network takes the Generator's outputs (waveforms) and estimates the Generator's latent code variables $c_1$ and $c_2$. Weights of both the Generator network and the Q-network are updated according to the Q-network's loss function: to minimize the distance between the actual one-hot vector ($c_1$ and $c_2$) used by the Generator and the one-hot vector estimated with a softmax in the Q-network's final layer using cross-entropy. This forces the Generator to output informative data.

The advantage of the ciwGAN architecture is that the network not only learns to output innovative data that resemble speech in the input, but also provides meaningful representations about data in an unsupervised manner. For example, as will be argued in Section , the ciwGAN network encodes reduplication as a meaningful category: it learns to assign a unique code for bare and reduplicated items. This encoding emerges in an unsupervised fashion from the requirement that the Generator output data such that unique information is retrievable from its acoustic outputs. Given the structure of the training data, the Generator is most informative if it encodes presence of reduplication in the code variables.

To replicate the results and to test learning of an identity-based pattern without binarization and without the requirement on the Generator to output informative data, we run an independent experiment on a bare WaveGAN architecture using the same training data. The difference between the two architectures is that the bare GAN architecture does not involve a Q-network and the latent space only includes latent variables uniformly distributed on the interval $(-1, 1)$.

and also propose a technique for latent space interpretability in GANs: manipulating individual variables to values well beyond the training range can reveal underlying representations of different parts of the latent space. We use this technique throughout the paper to evaluate learning of reduplication.

#  Reduplication in training data

The training data was constructed to test a simple reduplication pattern, common in human languages: partial CV reduplication found in languages such as Paamese, Roviana, Tawala, among others . Base items are of the shape C$_1$V$_2$C$_3$V$_4$ (C = consonant; V = vowel), e.g. /tala/. Reduplicated forms are of the shape C$_1$V$_2$C$_1$V$_2$C$_3$V$_4$, where the first syllable (C$_1$V$_2$) is repeated. The items were constructed so that C$_1$ contains a voiceless stop /p, t, k/, a voiced stop /b, d, g/, a labiodental voiced fricative /v/, and nasals /m, n/. The vowels V$_2$ and V$_4$ consist of / (), i, u/. C$_3$ consists of /l, , j/. All permutations of these elements were created. The stress was always placed on V$_2$ in the base forms and on the same syllable in reduplicated forms (\[\] $\sim$ \[\]). Because the reader of the training data was a speaker of American English, the training data is phonetically even more complex. The major phonetic effects in the training data include (i) reduction of the vowel in the unstressed reduplicated forms and in the final syllable (e.g. from \[\] to \[\]) and (ii) deaspiration of voiceless stops in the unstressed reduplication syllable (e.g. from to ). The training data includes two unique repetitions of each item and two repetitions of the corresponding reduplicated forms. Table illustrates the training data.

The training data also includes base forms C$_1$V$_2$C$_3$V$_4$ with the initial consonant C$_1$ being a fricative \[s\]. These items, however, always appear unreduplicated in the training data — the purpose of \[s\]-initial item is to test how the network extends the reduplicative pattern to novel unobserved data. All 27 permutations of sV$_2$C$_3$V$_4$ were included. To increase representation of \[s\]-initial words, four or five repetitions of each unique \[s\]-initial base were used in training.[^4] Altogether 132 repetitions of the 27 unique unreduplicated words with an initial \[s\] were used in training.

Sibilant fricative \[s\] was chosen as C$_1$ for testing learning of reduplication because its frication noise is acoustically prominent and sufficiently different from C$_1$s in the training data both acoustically and phonologically. This satisfies the requirement that a model learns to generalize to novel segments and feature values .[^5] In phonological terms, the model is tested on a novel feature (sibilant fricative or \[$\pm$strident\]; ) — the training data did not consist of any bare or reduplicated forms with other sibilant fricatives. To make the learning even more complex, voiceless fricatives (\[f, , \]) are altogether absent from the training data. All voiced fricatives except for \[v\] are absent too. Spectral properties of the voiced non-sibilant fricative \[v\] in the training data (and in Standardized American English in general) are so substantially different from a voiceless sibilant fricative \[s\] that we kept them in the training data. We excluded all items with initial sequences /ti/, /tu/, and /ki/ from the training data, because acoustic properties of these sequences, especially frication of the aspiration of /t/ and /k/, are similar to those of frication noise in /s/. Altogether 996 unique sliced items used in training were recorded in a sound attenuated booth by a female speaker of American English with a MixPre 6 (SoundDevices) preamp/recorder and the AKG C544L head-mounted microphone.

|                     |                                |     |
|:--------------------|:-------------------------------|:----|
| voiceless C$_1$     | C$_1$V$_2$C$_3$V$_4$           |     |
|                     | C$_1$V$_2$C$_1$V$_2$C$_3$V$_4$ |     |
| voiced C$_1$        | C$_1$V$_2$C$_3$V$_4$           |     |
|                     | C$_1$V$_2$C$_1$V$_2$C$_3$V$_4$ |     |
| C$_1$ = \[m, n, v\] | C$_1$V$_2$C$_3$V$_4$           |     |
|                     | C$_1$V$_2$C$_1$V$_2$C$_3$V$_4$ |     |
| C$_1$ = \[s\]       | C$_1$V$_2$C$_3$V$_4$           |     |
|                     | C$_1$V$_2$C$_1$V$_2$C$_3$V$_4$ | —   |

A schematic illustration of the training data in the International Phonetic Alphabet.

# CiwGAN

The Generator features two latent code variables, $c_1$ and $c_2$ and 98 uniformly distributed variables $z$ (Figure ). In the training phase, the two code variables ($c_1$ and $c_2$) compose the one-hot vector with two levels: \[0, 1\] and \[1, 0\]. This means that the network can encode two categories in its latent space structure that correspond to some meaningful feature about the data. The Q-network forces the Generator to encode information in its latent space. In other words, the loss function of the Q-network forces the Generator to output data such that the Q-network is effective in retrieving the latent code $c_1$ and $c_2$ from the Generator's acoustic outputs only. Nothing in the training data pairs base and reduplicated forms. There is no overt connection between the bases and their reduplicated correspondents. Yet, the structure of the data is such that given two categories, the most informative way for the Generator to encode unique information in its acoustic outputs is to associate one unique code with base forms and another with reduplicated forms. The Generator would thus have a meaningful unique representation of reduplication that arises in an unsupervised manner exclusively from the requirement on the Generator to output informative data.

To test whether the Generator encodes reduplication in latent codes, we train the network for 15,920 steps (or approximately 5,114 epochs) with the data described in Section . The choice of the number of steps is based on two objectives; first, the output data should approximate speech to the degree that allows acoustic analysis. Second, the Generator network should not be trained to the degree that it replicates data completely. As such, overfitting rarely occurs in the GAN architecture . The best evidence against overfitting in the ciwGAN architecture comes from the fact that the Generator outputs data that violate training distributions substantially (see Section below) . Despite these guidelines, the choice of number of steps is somewhat arbitrary (for discussion, see ).

We generate 100 outputs for each latent code \[0, 1\] and \[1, 0\] (200 total) and annotate them for presence or absence of reduplication. All annotations here and in other sections are performed by the author in Praat . Distinguishing unreduplicated from reduplicated is very salient; for less salient annotations, we provide waveforms and spectrograms (e.g. Figures and ).[^6]

There is a significant correlation between the two levels of latent code and presence of reduplication. Counts are given in Table . When the code is set to \[1, 0\], 78% of the generated outputs are base forms; when set to \[0, 1\], 60% of outputs are reduplicated (odds ratio = 5.27, $p<0.0001$, Fisher Exact Test). When the latent codes are set to \[0, 5\] and \[5, 0\], we get a near categorical distribution of bare and reduplicated forms. For \[5, 0\], the Generator outputs an unreduplicated bare form in 98% samples. For \[0, 5\], it outputs a reduplicated form in 87% outputs (odd ratio = 308.3, $p<0.0001$, Fisher Exact Test). These outcomes suggest that the Generator encodes reduplication in its latent codes and again confirm that manipulating latent variables well beyond training range reveals the underlying learning representations in deep convolutional networks (as proposed in ).

|     **Code** | **Bare** | **Redup.** | **% Redup.** |
|-------------:|:--------:|:----------:|:------------:|
| **\[1, 0\]** |    78    |     22     |     22%      |
| **\[0, 1\]** |    40    |     60     |     60%      |
| **\[5, 0\]** |    98    |     2      |      2%      |
| **\[0, 5\]** |    13    |     87     |     87%      |

Counts of bare and reduplicated (redup.) outputs when the latent codes $c_1$ and $c_2$ are set to \[1, 0\], \[0, 1\], \[5, 0\], and \[0, 5\].

## Interpolation

That the Generator uses latent codes to encode reduplication is further suggested by another generative test performed on interpolated values of the latent code. To test how exactly the relationship between the latent codes ($c_1$ and $c_2$) works, we created sets of generated outputs based on interpolated values of the code $c_1$ and $c_2$. We manipulate $c_1$ and $c_2$ from the value 1.5 towards 0 in increments of 0.125. For example, we start with \[1.5, 0\] and interpolated first to \[0, 0\] (e.g. \[1.375, 0\], \[1.25, 0\], etc.). From \[0, 0\] we further interpolate in increments of 0.125 to \[0, 1.5\] (e.g. \[0, 0.125\], \[0, 0.25\]). All other variables in the latent space are kept constant across all interpolated values. Each such set thus contains 25 generated samples. We generate 100 such sets (altogether 2500 outputs) and analyze each output. Out of the 100 sets, the output was either bare or reduplicated throughout the interpolated values and did not change in 55 sets. As suggested by Section and Table , the number of bare and reduplicated forms for each level rises to near categorical values as the variables approach values of 5.

In the 45/100 sets, the output changes from the base form to a reduplicated form at some point as the codes are interpolated. If the network only learned to randomly associate base and reduplicated forms with each endpoint of the latent code, we would expect base forms to be unrelated to reduplicated forms. For example, a base form \[\] could turn into reduplicated \[\]. An acoustic analysis of the generated sets, however, suggests that the latent code directly corresponds to reduplication. In approximately 25 out of 45 sets (55.6%) of generated outputs that undergo the change from base to a reduplicated form (or 25% of the total sets), the base form is identical to the reduplicated form with the only major difference between the two being the presence of reduplication (waveforms and spectrograms of the 25 outputs are in Figure ). This proportion would likely be even higher with a higher interpolation resolution (higher than 0.125) and because we do not count cases in which major changes of sounds occur besides the addition of the reduplication syllable (for example, if \[\] changes to \[\], we count the output as unsuccessful). In the remaining 20 outputs, several outputs undergo changes, where several segments or their features are kept constant, but the degree to which they differ can vary (e.g. \[\] $\sim$ \[\], \[\] $\sim$ \[\], \[\] $\sim$ \[\], or \[\] $\sim$ \[\]).

Under the null hypothesis, if the Generator learns to pair the base and reduplicated forms randomly, each base form could be associated with any of the unique 243 reduplicated forms at the probability of 1/243 (0.004). Even if we assume very conservatively that each base form could be associated with only each subgroup of reduplicated consonant (C$_1$; e.g. voiceless stops, voiced stops, \[m\], \[n\], \[v\]) disregarding the vowel and disregarding changes in the base, the probability of both forms being identical would still be at only 0.2 (for each of the five subgroups). In both cases, the ratio of identical base-reduplication pairs, while not categorical, is highly significant ($\text{CI} = [0.4,~0.7], p<0.0001$ for both cases according to Exact Binomial Test).

Figure illustrates how, keeping the latent space constant except for the manipulation of the latent code with which the Generator represents reduplication, the generated outputs gradually transition from the base forms \[\] and \[\] to the reduplicated forms \[\] and \[\].[^7] Other major properties of the output are unchanged.

This interpolative generative test again suggests that the network learns reduplication and encodes the process in the latent codes. By interpolating the codes we can actively force reduplication in the output with no other substantial changes in the majority of cases.

## Reduplication of unobserved data

To test whether the ciwGAN network learns to generalize the reduplicative pattern on unobserved data, we use latent space manipulation to force reduplication at the same time as presence of \[s\] in the output. Items with a \[s\] as the initial consonants (e.g. \[\]) appear only in bare forms in the training data. In Sections and , we established that the network uses the latent code ($c_1$ and $c_2$) to represent reduplication. Following and , we can force any phonetic property in the output by manipulating the latent variables well beyond the training range. Reduplication is forced by setting the latent code to values higher than \[0, 1\]. We can simultaneously force \[s\] in the output to test the network's performance on reduplication in unseen data.

To identify latent variables with which the Generator encodes the sound \[s\] in the output, we generate 1000 samples with randomly sampled latent variables, but with the latent code variables ($c_1$ and $c_2$) set at \[0, 1\] and \[1, 0\] (500 samples each with the same latent variable structure of the remaining 98 variables across the two conditions). We annotate outputs for presence of \[s\] for the two sets and fit the data to a Lasso logistic regression model in the *glmnet* package . Presence of \[s\] is the dependent variable coded as a success; the independent variables are the 98 latent variables uniformly distributed on the interval $(-1, 1)$ (for the technique, see ). Lambda is computed with 10-fold cross validation. Estimates of the Lasso regression model (Figure ) suggest that $z_{90}$ with the highest regression estimates is one of the variables with which the Generator encodes presence of \[s\] in the output. For a generative test providing evidence that Lasso regression estimates correlate with network's internal representations, see .

We can thus set $z_{90}$ to marginal levels well beyond the training range and the latent code ($c_1$, $c_2$) to levels well beyond \[0, 1\] in order to force reduplication and \[s\] in the output simultaneously. For example, when the latent code is set to \[0, 3\] (which forces reduplication in the output) and $z_{90}$ to 4 (forcing \[s\] in the output), the network outputs a reduplicated \[\] (among other outputs) even though items containing an \[s\] are never reduplicated in the training data. When the code is set to even higher number, \[0, 7.25\], and $z_{90}$ to 7, the network outputs \[\] in a different output. The spectrograms in Figure show a clear period of frication noise characteristic of a sibilant fricative \[s\], interrupted by a reduplicative vowel and followed by a repeated period of frication noise characteristic of \[s\].

In fact, at the values \[0, 7.25\], and $z_{90}= 7$, the network generates approximately 33 (out of 100 tested or 33%) outputs that can be reliably analyzed as reduplicated forms with initial sV- reduplication unseen in the training data. The other 67 outputs are reduplicated forms containing other C$_1$s or unreduplicated \[s\]-forms. No outputs were observed in which C$_1$ of the reduplication syllable and C$_1$ of the base would be substantially different. While all the cases when $z_{90}$ is manipulated involve a front vowel \[i\] in the base item, we can also elicit reduplication for other vowels. For example, we identify variable $z_4$ as corresponding to an \[s\] and a low vowel in the output (with the same technique as described for $z_{90}$ above but with presence of \[s\] as the dependent variable in the Lasso regression model). By manipulating $z_4$ to 9.5 (forcing \[\] in the output) and setting the latent codes to \[0, 7.5\], we get \[\] in the output (Figure ).

For comparison, the same L1 speaker of English who read the words in the training data read the reduplicated \[\] and \[\] which were not included in the training data. Figure parallels the generated reduplicated forms based on unobserved data (which were elicited by forcing \[s\] and reduplication in the output) and the recording of the same reduplicated form read by a human speaker. The spectrograms show clear acoustic parallels between the Generated outputs and the recording read by a human speaker (who read the words prior to computational experiments and did not hear or analyze the generated outputs).

# Replication: Bare WaveGAN

To test whether the learning of reduplicative patterns in GANs is a robust or idiosyncratic property of the model presented in Section , we conduct a replication experiment. We introduce two crucial differences in the replication experiment: we train the Generator without the requirement to produce informative data and without binary latent codes. We use the model in which features a "bare" GAN architecture for audio data: only the Generator and Discriminator networks without the Q-network. This architecture has the potential to inform us how GANs represent reduplicative patterns without an explicit requirement to learn informative data, i.e. without an explicit requirement to encode some salient feature of the training data in the latent space. The data used for training is the same as in the experiment in Section . We train the network for 15,930 steps or approximately 5,118 epochs, which is almost identical to the number of steps/epochs in the ciwGAN experiment (Section ).

##  Identifying variables

Testing the learning of reduplication in the bare GAN architecture requires that we force reduplication and presence of \[s\] in the output simultaneously. To identify which latent variables correspond to the two properties, we use the same technique as described in Section . We generate and annotate 500 outputs of the Generator network with randomly sampled latent variables. We annotate the presence of \[s\] and the presence of reduplication. The annotations are fit to a Lasso logistic regression (as in Section ): presence of reduplication or \[s\] are the dependent variables and each of the 100 latent $z$-variables are the independent predictors. Lambda values were computed with 10-fold cross validation. Regression estimates are given in Figure .

The plots illustrate a steep drop in regression estimates between the few latent variables with the highest estimates and the rest of the latent space. In fact, in both models, one or two variables per model emerge with substantially higher regression estimates: $z_{91}$ and $z_{5}$ when the dependent variable is <span class="smallcaps">presence of reduplication</span> and $z_{17}$ when the dependent variable is <span class="smallcaps">presence of</span> \[s\] in the output. We can assume the Generator network uses these two variables to encode presence of reduplication and \[s\], respectively.

It has been argued in that GANs learn to encode phonetic and phonological representations with a subset of latent variables. The discretized representation of continuous phonetic properties in the latent space appears even more radical in the present case. For example, in , presence of \[s\] as a sound in the output is represented by at least seven latent variables, each of which likely controls different spectral properties of the frication noise. In the present experiment, the Generator appears to learn to encode presence of \[s\] with a single latent variable, as is suggested by a steep drop of regression estimates after the first variables with the highest estimates. For a generative test showing that regression estimates correlate to actual rates of a given property in generated data, see . Such near-categorical cutoff is likely a consequence of the training data in the present case being considerably less variable compared to TIMIT (used for training in ). The network also represents an identity-based process, reduplication, with only two latent variables and features a substantial drop in regression estimates after these two variables. This discretized representation thus emerges even without the requirement of the Generator to output informative data.

In the replication experiment too, the Generator network outputs reduplicated forms for unobserved data when both reduplication and \[s\] are forced in the output via latent space manipulation, but significantly less so than in the ciwGAN architecture. When $z_{91}$ (forcing reduplication) and $z_{17}$ (forcing \[s\] in the output) are set to value $-8.5$, a higher level compared to the generated samples in the ciwGAN architecture (7 and 7.25), the network outputs only one reduplicated form with \[s\]-reduplication out of 100 generated outputs. By comparison, the proportion of the \[s\]-reduplication in the ciwGAN architecture is 33/100 – a significantly higher ratio ($\text{OR} = 48.1, p<0.0001$; Fisher Exact Test). When $z_{5}$ (forcing reduplication) is set to $-9.25$ and $z_{17}$ (forcing \[s\] in the output) to $-9.0$, the proportion of reduplicated \[s\]-items is slightly higher (4/100), but still significantly lower than in the ciwGAN architecture ($\text{OR} = 11.7, p<0.0001$; Fisher Exact Test). Despite these lower proportions of reduplicated \[s\] in the output, the bare GAN network nevertheless extends reduplication on novel unobserved data. Figure illustrates an example of a reduplicated \[s\]-item from the Generator network trained in the bare GAN architecture: \[\]. The spectrogram reveals a clear period of frication noise characteristic of an \[s\], followed by a reduplicative vowel period, followed by another period of frication.

# Discussion

We perform four generative tests to model learning of reduplication in deep convolutional networks: (i) a test of proportion of outputs when latent codes are manipulated to marginal values, (ii) a test of interpolating latent variables, (iii) a test of reduplication on unobserved data in the ciwGAN architecture, and (iv) a replication test of reduplication on unobserved data in the bare WaveGAN architecture. All four tests suggest that deep convolutional networks learn a simple identity-based pattern in speech called reduplication, i.e. a process that copies some phonological material to express new meaning. The ciwGAN network learns to encode a meaningful representation — presence of reduplication into its latent codes. There is a near one-to-one correspondence between the two latent codes $c_1$ and $c_2$ and reduplication. By interpolating latent codes, we cause the bare form to gradually turn into a reduplicated form with no other major changes in the output in the majority of cases. These results are close to what would be considered appearance of symbolic computation or algebraic rules.

Additional evidence that an approximation of symbolic computation emerges comes from the bare GAN experiment: there is a substantial drop in regression estimates after the first one or two latent variables with highest regression estimates, suggesting that even without the requirement to produce informative data, the network discretizes the continuous and highly variable phonetic feature — presence of reduplication — and uses a small subset of the latent space to represent this morphophonological property.

Finally, we can force the Generator to output reduplication at nearly categorical levels. When latent codes are set to marginal levels outside of training range (e.g. to \[5, 0\] or \[0, 5\]), the outputs are almost categorically unreduplicated or reduplicated (at 98 % for \[5, 0\]). shows that even higher values (e.g. 15) result in performance at 100% for a subset of variables. Not all aspects of the models in this paper are categorical (e.g. interpolation of latent codes does not always change an unreduplicated to a reduplicated form without other major changes). Improving performance on this particular task is left for future work. Inability to derive categorical processes has long been an argument against the connectionist approaches to language modeling. The results of this experiments add to the work suggesting that manipulating variables to extreme marginal values results in near categorical or categorical outputs (depending on the value) of a desired property .

In sum, three properties of rule-like symbolic representations emerge in deep convolutional network tested here: discretized representations, the ability to generate desired property by manipulating a small number of variables, and near categoricity for a subset of representations. These symbolic-like outcomes are facilitated by two inductive biases: the binary nature of latent codes and the requirement on the Generator to output informative data (forced by the Q-network). At least a subset of these properties also emerges in the bare WaveGAN architecture that lacks these biases, but at a reduced performance.

Encoding an identity-based pattern as a meaningful representation in the latent space emerges in a completely unsupervised manner in the ciwGAN architecture — only from the requirement that the Generator output informative data. Reduplicated and unreduplicated forms are never paired in the training data. The network is fed bare and reduplicated forms randomly. This unsupervised training approximates conditions in language acquisition (for hearing learners): the human language learner needs to represent reduplication and to pair bare and reduplicated forms from raw unlabeled acoustic data. The ciwGAN learns to group reduplicated and unreduplicated forms and assign a unique representation to the process of reduplication. In fact, the one-hot vector ($c_1$ and $c_2$) that the Generator learns to associate with reduplication in training can be modeled as a representation of the unique meaning/function that reduplication adds, in line with an approach to represent unique semantics with one-hot vectors (e.g. in ).

The paper also argues that deep convolutional networks can learn a simple identity-based pattern (copying) from raw continuous data and extend the pattern to novel unobserved data. While the network was not trained on reduplicated items that start with an \[s\], we were able to elicit reduplication in the output following a technique proposed in . First, we identify variables that correspond to some phonetic/phonological representation such as presence of \[s\]. We argue that setting single variables well above training range can reveal the underlying value for each latent variable and force the desired property in the output. We can thus force both \[s\] and reduplication in the output simultaneously. For example, the network outputs \[\] if we force both reduplication and \[s\] in the output; however, it never sees \[\] in the training data — only \[\] and other reduplicated forms, none of which included an \[s\].

Thus, these experiments again confirm that the network uses individual latent variables to represent linguistically meaningful representations . Setting these individual variables to values well above the training interval reveals their underlying values. By manipulating these individual variables, we can explore how the representations are learned as well as how interactions between different variables work (for example, between the representation of reduplication and presence of \[s\]). The results of this study suggest that the deep convolutional network is not only capable of encoding different phonetic properties in individual latent variables, but also processes as abstract as copying or reduplication.

One of the advantages of probing learning in deep convolutional neural networks on speech data trained with GANs is that the innovative outputs violate training data in structured and highly informative ways. The innovative outputs with reduplication of \[s\]-initial forms such as \[\] can be directly paralleled to acoustic outputs read by L1 speaker of American English that were absent from the training data. Acoustic analysis shows a high degree of similarity between the generated reduplicated forms and human recordings, meaning that the network learns to output novel data that are linguistically interpretable and resemble human speech processes even though they are absent from the training data. Thus, the results of the experiments have implications for cognitive models of speech acquisition. It appears that one of the processes that has long been held as a hallmark of symbolic computation in language, reduplication, can emerge in deep convolutional network without language-specific components in the model even when they are trained on raw acoustic inputs.

The present paper tests a simple partial reduplicative pattern where only CV is copied and appears before the base item. This is perhaps computationally the simplest reduplicative pattern. The training data are also highly controlled and recorded by a single speaker. We can use the well-understood identity-based patterns in speech with various degrees of complexity (longer reduplication, embedding into non-reduplicative patterns) to further test how inductive biases and hyperparameter/architecture choices interact with learning in deep convolutional networks. Finally, learning biases in the ciwGAN model can be (superficially) compared to learning biases in human subjects in future work. This paper suggests that the Generator provides informative outputs even if trained on comparatively small data sets (for a similar conclusion for other processes, see ). This means we can use the same training data to probe learning in CNNs and in human artificial grammar learning experiments (for a methodology, see ). While these comparisons are necessarily superficial at this point, they can provide insights into common learning biases between human learners and computational models.

# Acknowledgements

This work was supported by a grant to new faculty at the University of Washington. I would like to thank Ella Deaton for recording and preparing stimuli as well as anonymous reviewers and the Action Editor for useful comments on earlier versions of this paper.

# Appendix

[^1]:  proposes another approach that allows modeling reduplication. For a non-connectionist computational model of reduplication, see .

[^2]: The number of latent variables were adopted from and . Probing how the number of $z$-variables affects learning of speech representations is left for future work.

[^3]: For all details about the architecture, see .

[^4]: Items \[\], \[\], and \[\] each miss one repetition (four altogether).

[^5]: For an "across the board" generalization, requires that generalization occur to segments fully absent from the inventory. It is challenging to elicit reduplication of segments that are fully absent from the training data in the proposed models. Even in human subject experiments testing the "across the board" generalization, subjects need to be exposed to the novel segment at least as a prompt. In our case, the novel segment needs to be part of the training data, but only in unreduplicated forms.

[^6]: The code is available at <https://github.com/gbegus/fiwGAN-ciwGAN>. The generated data and checkpoints are available at <https://doi.org/10.17605/osf.io/zbjcp>.

[^7]: The exact vowel quality estimation in the generated outputs is challenging, especially in short vocalic elements of reduced vowels in the reduplicative syllables. For this reason, we default transcriptions to a \[\].


abstract: A short review paper for the quantum anomalous Hall effect. A substantially extended one is published as *Adv. Phys.* **64**, 227 (2015).
author: Hongming Weng; Xi Dai; Zhong Fang
date: Aug. 24, 2013
title: From Anomalous Hall Effect to the Quantum Anomalous Hall Effect

In 1879, Edwin H. Hall discovered that when a conductor carrying longitudinal current was placed in a vertical magnetic field, the carrier would be pressed to against the transverse side of the conductor, which led to the observed transverse voltage. This is called Hall effect (HE) , which is a remarkable discovery, in spite of that the understanding of it was difficult at that time since electron was to be discovered 18 years later. After that, as we know, HE is well understood and it is the Lorentz force experienced by the moving electrons in magnetic field pressing them against the transverse side. The HE is now widely used as to measure the carrier density or the strength of magnetic fields.

In 1880, Edwin H. Hall further found that this "pressing electricity" effect can be larger in ferromagnetic (FM) iron than in non-magnetic conductors. The enhanced Hall effect comes from the additional contribution of the spontaneous long range magnetic ordering, which can be observed even without applying external magnetic field. To be distinguished from the former one, this is termed as anomalous Hall effect (AHE) . Though HE and AHE is quit similar to each other, the underlying physics are much different since there is no orbital effect (Lorentz force) when external magnetic field is absent in AHE.

The mechanism of AHE has been a enigmatic problem since its discovery, and it lasted for almost a century. The AHE problem involves concepts deeply related with topology and geometry that have been formulated only in recent years after the Berry phase being recognized in 1984 . In hindsight, Karplus and Luttinger  provided a curtail step in unraveling this problem as early as in 1954. They showed that moving electrons can have an additional contribution to its group velocity when an external electric filed is applied. This additional term, dubbed "anomalous velocity", which is contributed by all occupied band states in FM conductors with spin-orbit coupling (SOC), can be non-zero and leads to the AHE. Therefore, this contribution depends only on the band structure of perfect periodic Hamiltonian and is completely independent of scattering from impurities or defects (therefore called intrinsic AHE), which makes it hard to be widely accepted before the concept of Berry phase being well established. In a long time, two other "extrinsic" contributions had been thought as the dominant mechanisms that give rise to an AHE. One is the skew scattering  from impurities caused by effective SOC and the other one is the side jump  of carriers due to different electric field experienced when approaching and leaving an impurity. The controversy arises also because it is hard to make quantitive comparison with experimental measurements. The unavoidable defects or domains in samples are complex and hard to be treated quantitively within any extrinsic model and the contributions from both "intrinsic" and "extrinsic" mechanisms co-exist.

In 1980, K. von Klitzing *et al.* made remarkable discovery of quantum Hall effect (QHE) . He found that, with the increase of external magnetic filed, the Hall conductivity exhibits a serials of quantized plateau, and at the same time, the longitudinal conductivity becomes zero, i. e., the sample bulk becomes insulating when Hall conductivity is quantized. In the QHE, electrons constrained in two-dimensional (2D) samples, are enforced to change its quantum states into new ones, namely the Landau energy levels, under highly intensive magnetic field. The originally free-like conducting electrons start to make cyclotron motion. If the magnetic field is strong enough, such cyclotron motions will form full circles, which makes electrons localized in the bulk and the sample becomes insulating. While along the edges of the 2D system, the circular motion enforced by the magnetic field can not be completed due to the presence of the edges, which enforce the electrons to travel in one way forming so called edge states. The electrons in such state can circumambulate defects or impurities on their way "smartly". Therefore, current carried by these electrons is dissipationless and conductance is quantized into unit of $e^2/\hbar$ with quantum number corresponding to the number of edge states. Such fascinating quantum state and physical phenomena are highly interesting and impact the whole field of physics, because this is a new state of condensed matters and it should be characterized by the topology of electronic wave-function . This topological number is given by D. J. Thouless, M. Kohmoto, M. P. Nightingale and M. den Nijs  in 1982 and is called as the TKNN number or the first Chern number, which has direct physical meaning as the number of edge states or the quantum number of Hall conductance.

After reaching this point, one may immediately ask the question: Can we have the quantum version of AHE, similar to the QHE? Unfortunately, this question was irrelevant at that historical moment, because we even did not know yet the fundamental mechanism of AHE then. Nevertheless, in 1988, F. D. M. Haldane proposed  that a QHE without any external magnetic field is in principle possible. Although this proposal is very simple and has nothing to do with either the AHE or the SOC, his idea play important roles for many of our nowadays studies, such as the topological insulators and the quantum anomalous Hall effect (QAHE), because he pointed out the possibility to have a novel electronic band structure of perfect crystal, which carry non-zero Chern number even in the absence of external magnetic field.

The underlying physics of AHE, in particular, the topological nature of intrinsic AHE was not fully appreciated only until the early years of 21 century. In a series of papers, (i.e, Jungwirth, et.al (2002) ; Onoda, et.al. (2002) ; Fang, et.al (2003) ; Yao, et.al. (2004) ), it was discovered that the intrinsic AHE can be related to the Berry phase of the occupied Bloch states. The so-called "anomalous velocity" is originated from the Berry-phase curvature, which can be regarded as effectively magnetic field in the momentum space, and thus modifies the equation of motion of electrons, leading to the AHE. This effective magnetic field can be also traced back to the band crossings and the magnetic monopoles in the momentum space , which is now called Weyl nodes in the Weyl semimetals . The quantitative first-principles calculations for SrRuO$_3$  and Fe , in comparison with experiments, suggested that intrinsic AHE actually dominates over extrinsic ones. The presence of SOC and the breaking of time reversal symmetry (due to the FM ordering) are crucially important, because otherwise the Berry phase contribution will be prohibited by symmetry. What makes those understandings unique is that, like the QHE, the intrinsic AHE is now directly linked to the topological properties of the Bloch states. Up to this point, it is now proper to ask the question: Can the AHE also be quantized like its cousin HE? If it is realized, we will have a kind of QHE without magnetic field, although it is already much different from Haldane's original speculation in the sense that SOC must play important roles here. Nevertheless, from the applicational point of view, the realization of QAHE will certainly stimulate the wide usage of such novel quantum phenomena in future technology, in particular, the dissipationless edge transport without magnetic field.

To reach this goal, there is still a big step to be overcome. The QAHE requires four necessary conditions: (1) the system must be 2D; (2) Insulating bulk; (3) FM ordering; and (4) non-zero Chern number. It may be easy to satisfy one or two conditions, but hard to realize all of them simultaneously. Fortunately, the recently rapid progresses in the studies of topological insulators (TI)  make the challenge of realizing QAHE possible. The TI is a new state of quantum matter, which is characterized by the $Z_2$ topological number  and is protected by the time-reversal symmetry (TRS), in contrast to the QHE and QAHE, where the TRS must be broken. However, an important view is that the 2D topological insulator and the quantum spin Hall effect (QSHE)  can be effectively viewed as two copies of distinct QAHE states which are related by the TRS, in other words the 2D QSHE state can be viewed as the time-reversal invariant version of QAHE state. Given the known material realizations of 2D and 3D TI , it is natural to start from those systems and try to break the TRS in order to achieve the QAHE. Following this strategy, several possibilities are proposed theoretically. Qi, et.al.  first pointed out that gapping the Dirac type surface states of 3D TI by FM insulating cap-layers may produce the QAHE, although their arguments are not concrete. Later on, the "band inversion" picture and the experimental observation of QSHE  inspired the idea of realizing QAHE by transition metal (Mn) doped HgTe quantum well structure . Unfortunately, in such case, the Mn local moments do not order ferromagnetically. In 2010, Yu, et.al,  predicted that when a topological insulator Bi$_2$Se$_3$ or Bi$_2$Te$_3$ is made thin and magnetically doped (by Cr or Fe), the system should order ferromagnetically through the van Vleck mechanism, and exhibit the QAHE with a quantized Hall resistance of $h/e^2$ — a proposal that was finally achieved experimentally by Chang, et.al.  after great efforts.

While this is not the end of the story, instead, we believe this success will inspire more extensive researches on QAHE. Two important issues become the natural directions for the future studies: (1) how to increase the temperature range of QAHE (it is now observed only in the tens mK range); (2) how to realize the higher plateau with Chern number larger than 1. There are several other proposals worth trying. HgCr$_2$Se$_4$ has been predicted to be a Weyl semimetal, and its quantum well structure with proper thickness would exhibit QAHE with Chern number being 2 , different from the Cr-doped Bi$_2$Te$_3$ family thin film. The advantage of this proposal is that HgCr$_2$Se$_4$ is a chemically pure and stable compound with known bulk Curie temperature higher than 100K. One similar proposal is GdBiTe$_3$ , the thin film of which have one edge state contributing to QAHE. Garrity and Vanderbilt  proposed that Au, Pb, Bi, Tl, I, and etc heavy-element layers on the surface of magnetic insulators may contribute to large AHE and even QAHE. With the further material breakthroughs, we have strong reason to expect that the QAHE may someday find its places in our electronic devices.

This work was supported by the National Science Foundation of China and by the 973 program of China.