Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep neural networks have achieved state-of-the-art performance in various fields, but they have to be scaled down to be used for real-world applications. As a means to reduce the size of a neural network while preserving its performance, knowledge transfer has brought a lot of attention. One popular method of knowledge transfer is knowledge distillation (KD), where softened outputs of a pre-trained teacher network help train student networks. Since KD, other transfer methods have been proposed, and they mainly focus on loss functions, activations of hidden layers, or additional modules to transfer knowledge well from teacher networks to student networks. In this work, we focus on the structure of a teacher network to get the effect of multiple teacher networks without additional resources. We propose changing the structure of a teacher network to have stochastic blocks and skip connections. In doing so, a teacher network becomes the aggregate of a huge number of paths. In the training phase, each sub-network is generated by dropping stochastic blocks randomly and used as a teacher network. This allows training the student network with multiple teacher networks and further enhances the student network on the same resources in a single teacher network. We verify that the proposed structure brings further improvement to student networks on benchmark datasets. Deep neural networks (DNNs) have achieved state-of-theart performances on complex tasks like computer vision (He et al. 2016), language modeling (Jozefowicz et al. 2016), and machine translation. Moreover, they surpass human ability in several fields including image classification (He et al. 2016), the go game, voice generation (Oord et al. 2016), and so on. Despite their superior performance, it is difficult to use DNN-based models because of limited memory and computational resources in the embedded systems. To deal with this problem, many studies have been done to make DNNs smaller but efficient to be applicable in resource limited cases. One of them is knowledge transfer (KT), which train a smaller network with the information of large model's information. Knowledge The primary goal of this paper is to make a single teacher network to behave as multiple teacher networks. Since multiple teacher networks provide various outputs on a given input, they can provide more extensive knowledge than a single teacher network does. It has been shown that student networks improve further with multiple teacher networks which are used as an ensemble or separately (Hinton, Vinyals, and Dean 2015; You et al. 2017; Zhang et al. 2018). However, using multiple teacher networks is a resource burden and delays the training process. In this work, we propose to add stochastic blocks and skip connections to a teacher network. In doing so, we can get the effect of multiple teacher networks in the same resource of single teacher network. A stochastic block is a block that falls with a fixed probability in the training phase and weighted by its survival probability in the inference phase. Skip connections make huge number of paths in the network and function as memory which link the information of previous parts and later parts even if stochastic blocks drop. In the training phase, different sub-networks are generated ing from stochastic drop in the teacher network for each batch. The sub-networks still have reliable performances since there still exist valid paths. Each sub-network becomes a teacher network for each batch, so the student network is trained with multiple teacher networks in the entire training phase. Figure 1 is example of sub-networks generated by dropping one block each from a network with the proposed structure. The networks consists of 3 blocks and f i, Id represents the ith block of the network (i ∈ 1, 2, 3) and an identity block generated by a skip connection respectively. Red arrows in the figure mean that the outputs of the blocks are 0. In Figure 1, even if one block drops, each subnetwork still has 4 valid paths of 8 total paths. We observe that: (i) multiple teacher networks are generated from a single teacher network with no more resources; (ii) generated networks provide different knowledge to a student network; (iii) the performances of student networks improve with the help of a teacher network of the proposed structure. We succeeded in training the student network to perform better than the ones with the same architecture trained by the knowledge transfer methods (KD) (Hinton, Vinyals, and Dean 2015), attention transfer (AT) (Zagoruyko and Komodakis 2016a), and mutual learning (ML) (Zhang et al. 2018) ) over CIFAR-100 (Krizhevsky, Hinton, and others 2009) and tinyimageNet (Russakovsky et al. 2015) datasets. The rest of this paper is organized as follows. First, we review recent studies related to our work. Then, we demonstrate the proposed scheme with details. After this, we present experiments and discuss the . Finally, summary and concluding remarks are given in the . Knowledge transfer of neural networks has been proposed over a decade ago (Buciluǎ, Caruana, and Niculescu-Mizil 2006) but has recently received much attention with some intuitions and a generalized approach (Hinton, Vinyals, and Dean 2015). There, softened outputs of a teacher network are used to transfer knowledge to a student network. They demonstrate that the softened outputs of a teacher network provide a student network with additional supervision and prevent the student network from overfitting. Later, distillation has been applied in transferring knowledge from powerful and easy-to-train networks to small but hard-to-train networks (Romero et al. 2014). Romero et al. suggest intermediate outputs of teacher networks to be used as hints for student networks. An attention-based distillation method makes use of attention maps of teacher networks which are made from feature maps (Zagoruyko and Komodakis 2016a). To transfer knowledge while avoiding direct mimicry, (Yim et al. 2017) exploits flows calculated by Gram matrix of feature maps from two layers of a teacher network, then a student network is trained to mimic the flows of the teacher network. Recently, mutual learning (Zhang et al. 2018) suggests a new paradigm of bidirectional knowledge transfer. All networks in mutual learning are not fixed and exchange knowledge unlike conventional teacher-student paradigm where teacher networks are fixed and student networks only get knowledge. Student networks can be improved further with the help of multiple teacher networks (You et al. 2017). The dissimilarity between teacher networks provide extensive knowledge to a student network and help to further enhance the student network. Similarly, (Zhang et al. 2018) shows that a neural network can be further improved with the help of multiple neural networks for vision tasks such as image classification and person re-identification. Also, (Chebotar and Waters 2016) shows that multiple teacher networks are more helpful than a single teacher in speech recognition. Most of the distillation methods improve the performance of student networks with multiple teacher networks, but deploying them is demanding due to additional resources. Instead of directly using multiple teacher networks, (Sau and Balasubramanian 2016) suggest perturbing the outputs of a teacher network to get the effect of multiple teacher networks. However, perturbing outputs with noise can be problematic as it changes the values of the outputs so that corrupted knowledge of the teacher network is transferred. In our proposed structure, multiple networks of valid paths are generated (see Figure 1) so that reliable and various outputs are transferred to the student network and provide flexible knowledge. In reinforcement learning, encouraging the policy to have an output distribution with high entropy has been used to improve exploration. This prevents the policy from converging early and leads to improved performance (Williams and Peng 1991; Mnih et al. 2016). Also, penalizing confident outputs (Pereyra et al. 2017) and smoothing label (Szegedy et al. 2016) are proved to help the training of a deep neural network. Regularizing the high confident outputs helps training the deep neural network since it prevents over-fitting of the network and a big difference between values of outputs so that the adaptivity of the network increases. In the same vein, high confident outputs of a teacher network are challenging for student networks to learn. In ML (Zhang et al. 2018), it has been shown that the ensemble of multiple networks is a worse teacher than the individual networks. Individual networks provide higher entropy outputs than the ensemble, so that the salient secondary values in the outputs can be more helpful generalizing student networks. To get the effect of multiple teacher networks from a single teacher network, we propose to add stochastic blocks and skip connections to the teacher network. In this section, first, we explain in detail how to change the structure of the teacher network to make multiple sub-networks. Then, we demonstrate that multiple sub-networks can be used as multiple teacher networks. For ResNet (He et al. 2016) and Wide ResNet (Zagoruyko and Komodakis 2016b), they consist of blocks and contain skip connections. For MobileNet (Howard et al. 2017), we grouped a depth-wise convolution and a point-wise convolution as one block and add skip connections from each input of the block to the corresponding output. Skip connections in residual networks prevent vanishing gradient problem, so that deeper networks can be trained well. In other respect, skip connections let a residual network to be viewed as an ensemble of multiple paths of different lengths (Veit, Wilber, and Belongie 2016). When we set ith block of a residual network as f i, then the output (o i+1) of the (i + 1) th block is expressed as follows. Since there are two paths from a previous output to the next output, if there are n blocks in the network, 2 n paths exist from the input layer to the output layer. It has been shown that, to some extent, changing the structure of a residual network do not harm the performance much (Veit, Wilber, and Belongie 2016). Especially, deleting blocks of residual networks does not harm the performance much. This is because there still exist valid paths even if some blocks of a residual network drop (if k blocks are dropped from n blocks, valid 2 n−k paths still exist). Therefore, when a neural network consists of blocks and contains skip connections, multiple neural networks with adequate performances are generated by dropping blocks randomly. To implement this idea in training phase, we set blocks of neural networks to be stochastic as in. Since initial blocks extract low-level features that will be used by later blocks, we choose linear decay mode to set the survival probability of each block. P end denotes the survival probability of the last block and p i survival denotes that of the ith block expressed as where N is the number of total blocks and i = 0 ∈ {0, 1, ..., N − 1}. P end implies a trade-off between quantity and quality of sub-networks. If p end is high, each generated sub-network will have longer length, so the performance will be better than sub-networks with shorter lengths. However, high p end generates less sub-networks. In the opposite case, more subnetworks are generated but each performance can be a bit lower. Optimal p end seems different for teacher and student pairs. We tried p end ranging [0.5, 0.9] with interval 0.1 and choose p end that improves student networks most for each teacher and student pair. One might wonder if sub-networks can play the role of teacher networks and provide independent knowledge so that student networks get sufficient knowledge. For a residual network of 110 layers, it has been shown that subnetworks generated by dropping some blocks show competent performance and are independent each other (Veit, Wilber, and Belongie 2016). Convolutional neural networks (CNNs) based on residual networks will probably have the same characteristic, however, other networks like mobile network are not guaranteed to generate reliable sub-networks with the proposed structure. To verify the efficacy, we show the accuracy when each block is dropped from pre-trained networks for three kinds of networks. Figure 2 is the accuracy when each block drops from residual network of 32 layers, mobile network, and wide residual network 28-10. In Figure 2, sto and basic represents networks with the proposed structure and original networks respectively. Sto networks are stronger against dropping blocks than basic networks, so we pre-train teacher networks with the proposed structure. It seems that dropping initial blocks of mobile network and 4th block of wide resnet 28-10 degrades the performance significantly. To observe the impact of such blocks that are fatal to drop, we compare cases where the blocks drop or does not drop like other blocks in ablation study. The performances of sub-networks lag behind the original network. However, they sometimes predict correctly while the original network does not. Also, they generate outputs with high entropy which are easier for student networks to learn (see Figure 3). It is known that regularizing a neural network to be less confident improves performance (Pereyra et al. 2017). Similar are also observed in deep mutual learning. In (Zhang et al. 2018), they show that using an ensemble of n networks as a teacher is less helpful than using n individual networks as n teachers. This is because the ensemble makes the outputs have low entropy, which means that the secondary values of outputs becomes small. The secondary values are salient cues in transferring knowledge as it provides important information like relations between classes. Dropping blocks of the teacher network is analogous to using individual networks instead of the ensemble of them. Thus, sub-networks can provide student networks with rich knowledge. Also, knowledge of the original network is fully utilized as all the blocks of the network are used in generating sub-networks in the end. Generated sub-networks share considerable parts of the original network but provide different knowledge to a student network. The degree of the difference is similar to that of individual neural networks. We confirm the similarity with resnet 32 and attach the related table in the appendix. We apply the proposed method to other distillation techniques, KD, AT, and ML. To apply to KD and AT, the teacher network is changed to have skip connections and stochastic blocks and other settings are not changed. In mutual learning, the notions of teacher and student vanish since both networks give and take knowledge each other. But for convenience, we denote a network with large capacity as a teacher network and the other as a student network. The teacher network is changed into proposed structure as in KD and AT. To apply the proposed structure to mutual learning, both networks should be pre-trained since teacher networks are not fixed. If the networks are not pre-trained, they cannot be improved because of the stochastic property of the teacher network. Let's assume a situation when both networks are not pre-trained. At the beginning of training process, subnetworks of a teacher network are randomized. In mutual training, each sub-network and a student network exchange knowledge. However, since a different sub-network is used each time, for many epochs, the student gets random knowledge from randomized sub-networks so that it does not improve. Also, sub-networks are not optimized due to the disturbing knowledge from the student network. In our simulation, multiple sub-networks are not used at the same time but one sub-network generated by stochastic drop is used as a teacher network for each batch. We evaluate the proposed method with two datasets -CIFAR-100 (Krizhevsky, Hinton, and others 2009) and tiny imagenet (Russakovsky et al. 2015). CIFAR-100 dataset consists of 32 × 32 RGB color images drawn from 100 classes, which are split into 50, 000 train and 10, 000 test images. Tiny imagenet dataset is a down-sampled version of ImageNet dataset. It consists of 64 × 64 RGB color images drawn from 200 classes, which are split into 100, 000 train and 10, 000 test images. For CIFAR-100, we normalize each image and augment the train images. The data augmentation includes horizontal flips and random crops from image padded by 4 pixels on each side, filling missing pixels with reflections of original image. Each network is trained for 200 epochs with batch size of 128 and learning rate which is decreased at every 60 epochs. For tiny imagenet, we simulate with the pure dataset without augmentation. Each network is trained for 100 epochs with batch size of 128 and learning rate which is decreased at every 40 epochs. Stochastic gradient descent optimizer with momentum of 0.9 is used for the whole simulation. The initial learning rate is 0.01 for ML case and 0.1 for the other cases. 4 CNNs are used -wrn, resnet, mobilenet, and vgg net (Simonyan and Zisserman 2014). CNNs are modified to the proposed structure when they are used as teacher networks. All the in the simulation are averaged over 3 times. Here, we present simulation of knowledge transfer methods on CIFAR-100. Table 1 is the simulation of KD and KD with the proposed structure. As you can see in Table 1, we confirm that the proposed structure further improves performances of student networks on KD. In case of (WRN 28-10, ResNet 32) pair, the accuracy of ResNet 32 trained with the proposed structure improves more than 5% compared to when ResNet 32 is trained the pure WRN 28-10. Table 2 is the simulation of AT and AT with the proposed structure. In AT, attention maps of teacher and student networks should have same spatial size. So, we used residual networks and wide residual networks to fit the spatial size conveniently. Attention maps are made by square sum via channel axis and l2 normalization. The proposed structure show further improvement over the pure AT method. We confirm that the proposed structure improve student networks further with AT method in all the pairs. Table 3 is the simulation of ML and ML with the proposed structure. Only teacher networks are change to the proposed structure, as mentioned in the previous section. And a teacher networks and a student network exchange knowledge each other. The network pairs in Table 3 are same with those of the paper (Zhang et al. 2018). The proposed structure still show further improvement in peer learning paradigm, so both networks are improved further. Here, we present simulation of knowledge transfer methods on tiny imagenet. Table 4, 5 are the simulation of KD, AT, and proposed structure. As the simulation on CIFAR-100, the proposed structure improves student networks generally, but there exist one pair each for KD, AT that the student network is not improved. In Figure 2, when some blocks drop, then, performances of neural networks drop significantly. The blocks are the 4th block of wrn 28-10 and 1st to 6th blocks of mobilenet. We name these blocks significant blocks. Sub-networks generated by dropping significant blocks have low performance so that the networks might not be adequate teacher networks. Hence, we observe if student networks improve further, when sub-networks generated by dropping the other blocks except the significant blocks are used as teacher networks. We use KD and CIFAR-100 dataset for (wrn 28-10, resnet 32) and (mobilenet, resnet 32) pairs. In Table 6, partial means that significant blocks do not drop and full means that all the blocks drop stochastically in training phase. The show that using more teacher networks is more helpful improving a student network even if some of them do not perform well. This is in line with the of ML (Zhang et al. 2018) where larger network still benefits from being trained together with a smaller network. In this work, we propose to change the structure of a teacher network to get the effect of multiple teacher networks in the same resource of one teacher network. In our proposed structure, we obtain multiple teacher networks without additional resource so that compact networks improve further than those trained from conventional transfer methods. The proposed structure can be easily applied to other transfer methods and tasks, e.g. segmentation or object detection.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

The goal of this paper is to get the effect of multiple teacher networks by exploiting stochastic blocks and skip connections.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Emoji suggestion systems based on typed text have been proposed to encourage emoji usage and enrich text messaging; however, such systems’ actual effects on the chat experience remain unknown. We built an Android keyboard with both lexical (word-based) and semantic (meaning-based) emoji suggestion capabilities and compared these in two different studies. To investigate the effect of emoji suggestion in online conversations, we conducted a laboratory text-messaging study with 24 participants, and also a 15-day longitudinal field deployment with 18 participants. We found that lexical emoji suggestions increased emoji usage by 31.5% over a keyboard without suggestions, while semantic suggestions increased emoji usage by 125.1%. However, suggestion mechanisms did not affect the chatting experience significantly. From these studies, we formulate a set of design guidelines for future emoji suggestion systems that better support users’ needs. forms of text-based computer-mediated communication (CMC) lack nonverbal expressions like vocal tones, facial expressions, and gestures that are useful in face-to-face conversations. However, several studies have shown that emojis can facilitate affective communication and serve a non-verbal function in text conversations. Emojis are already widely used in text-based CMC, with nearly every instant messaging platform supporting their entry. Five billion emojis were sent per day on Facebook Messenger in 2017, and half of all Instagram comments included an emoji as of mid-2015. Many mobile keyboards offer emojis as a set of pictographic Unicode characters. As there is a large and growing set of emojis, manually searching for and selecting emojis can be a tedious task interrupting the flow of text entry. Commercial products that automatically suggest emojis have helped the emoji entry process become more seamless. These products usually come in two variations-lexical and semantic suggestions-as shown in Table 1. With lexical suggestions (e.g., Gboard), relevant emojis appear in a candidate list based on recent keywords typed by the user. With semantic suggestions (e.g., Dango; Figure 1 ) proposed emojis are based on the meaning of the message's content rather than on specific keywords. Although emojis themselves are known to enrich conversations, the role that different emoji suggestion systems play has not been explored. Instead, prior work on suggestion systems has focused on retrieval precision and recall. But how do different suggestion mechanisms influence emoji usage? How do they differ in terms of usability? How do they influence the chat experience, specifically the engagement and the clarity of conversations? To investigate these questions, we implemented a keyboard capable of offering both lexical and semantic emoji suggestions. We first evaluated the performance of the two suggestion mechanisms with an online study. The I enjoyed the fish tonight very much! I love him but he just ignored me... I'm tired of "happy birthday" Table 1. Examples of lexical and semantic emoji prediction. With lexical prediction, the suggested emojis are related to the literal meaning of certain keywords. With semantic prediction, the suggestions focus on the meaning of the sentence. confirmed that semantic suggestions were perceived as being more relevant than lexical suggestions. We then conducted an in-lab study with pairs of participants using three emoji suggestion mechanisms (no suggestions, lexical suggestions, and semantic suggestions). We also conducted a 15-day field deployment to evaluate emoji usage in real-world settings. We found that emoji suggestion systems increased emoji usage overall, with users picking more emojis via semantic suggestions versus lexical suggestions or no suggestions. Users also felt that semantic emoji suggestions were more relevant than lexical emoji suggestions. Although semantic suggestions received more positive feedback in terms of usability than lexical suggestions, neither had a significant effect on the participants' perceived chat experience overall. The contributions of this work are: from an online study comparing the perceived relevance of lexical and semantic suggestions; from an in-lab study comparing emoji suggestion mechanisms within the mobile chat experience; from a longitudinal field deployment that tracked realistic usage of emoji suggestion systems; and design guidelines of emoji suggestion systems based on the findings from our studies. Emoji-related research has become more prominent as emojis have grown in number and popularity. In this section, we review related work from three different areas: emoji usage and its effects in online communication, emoji entry techniques, and the use of machine learning for producing semantic emoji suggestions. As Unicode character pictographs, emojis are treated similarly to other characters in text-based applications. In fact, emojis can even be used in text-only locales such as URLs. That being said, emojis represent richer information than plain text and are easier to share than images, giving emojis certain advantages over other forms of communication. 1 https://emojipedia.org/stats/ Emojis usage has steadily increased since emojis were introduced to the Unicode Standard in 2009. According to a report by Swiftkey in 2015, their users inputted over one billion emojis in a four-month period. Although over 800 emojis were available to users during that time, traditional "face" emojis (e.g.,) comprised nearly 60% of all emojis sent. Roughly 70% of the messages containing emojis expressed a positive emotion, and only 15% of the messages expressed a negative emotion. Jain et al. found that emojis are used to convey all kinds of emotions, and the number of emojis used in a message could determine the arousal of the sender. They also found that emoji combinations could be used to convey more complex expressions (e.g., meaning, "I'm relaxing and playing soccer"). Cramer et al. conducted an online survey with 228 respondents, finding three major reasons for why people use emojis: to provide additional emotional or situational information, to change the tone of a message, and to engage the recipient and maintain their relationship. Although every emoji has an intended definition, people also use emojis in highly personalized and contextualized ways to create "shared and secret uniqueness". Wiseman and Gould cited an example where a couple used the pizza emoji to express love because they both loved eating pizza. There is no doubt that emojis extend and enrich the way people express themselves in text-based CMC. Our current work focuses on the role that suggestion mechanisms play in facilitating such expressions. Pohl et al. provide a thorough review of emoji entry techniques. The most common entry method on current commercial keyboards is grouped enumeration, wherein users can scroll through different categories to select their emojis. However, as there are over 2,800 emojis, 1 so visually searching and selecting emojis is a tedious process. EmojiZoom displays all emojis at once, requiring users to zoom to select one. However, this method still fails to scale as the number of emojis increases. Querying techniques, such as text search or sketching, are implemented in many keyboards like Gboard. Users can search for emojis by sketching them or typing their intended meaning (e.g., "happy birthday" for a cake emoji). Such techniques require users to have a target emoji in mind, and the process is slow. Suggestion-based input methods have become popular in recent years. Lexical suggestions are offered by keyboards like the Apple iOS 11 keyboard. However, the suggestions do not work for all possible keywords, since keywords must be defined beforehand. For example, the pear emoji Figure 1. The semantic emoji suggestion application Dango. When text is typed, Dango pops up a suggested emoji based on semantic message content. The user can tap on an icon to see more options. appears in Gboard's suggestion list when "pear" is typed, but it disappears if "pears" is typed. A relatively new emoji suggestion technique that appears in products like Dango uses semantic information. Instead of relying on keywords, semantic suggestion offers emojis based on the sentiment of the whole message. This mechanism often provides affective emojis like faces. Google deployed a similar system called Smart Reply; rather than focusing on suggestions based on input, SmartReply auto-generates replies with emojis based on the context of the conversation. To suggest emojis using semantics, emojis must be linked with the meanings of typed messages. Our keyboard implementation relies on a method from prior work called DeepMoji by Felbo et al.. The implementation of their model used in this paper can be found on GitHub. There were several limitations of Felbo et al.'s approach. First, the model was built for sentiment classification tasks, so most of the emojis it predicted were related to the messages' emotions rather than their meanings. If a user typed "happy birthday," for example, the predicted emojis would be happy face emojis rather than the birthday cake emoji . Second, the model was trained to handle only 64 emojis. Although those emojis were the most common on Twitter, many of them were faces rather than objects. Nevertheless, our goal is to examine the effects of different types of suggestion mechanisms rather than improving on the mechanisms themselves, so we did not extend the output emoji set beyond the original 64. We built our Android keyboard using the open source project AnySoftKeyboard 3. The keyboard interface is shown in Figure 3. The keyboard uses the default auto-correction mechanism, but the word-suggestion feature is replaced with emoji suggestions. Users can enter special characters or numbers by tapping the upper-left button; they can enter emojis by tapping the lower-left button. The overall text entry interaction of the keyboard is shown in Figure 4. As a user types in the text box, the keyboard provides word suggestions in the candidate list. When the user finishes typing a word, the keyboard suggests emojis instead of words in the candidate list. If the user picks an emoji from the list, it is inserted at the end of the message. The suggestion varies based on the mechanism in use. With semantic suggestion, the keyboard always presents five emojis after the user finishes typing a word. Suggestions are generated using the DeepMoji model running on a remote server. The keyboard sends an HTTP POST request to the server each time the user finishes typing a word, and the server returns the top-five related emojis. (The amount of information transmitted is small and there were no latency concerns in our implementation or studies.) With lexical suggestion, the keyboard suggests emojis only if the keyword list contains the last-typed word. If no emoji matches the last-typed word, the keyboard presents the most recent suggestions. For example, if the user types "football field," the keyboard will continue to suggest the football emoji because there is no lexical match with the word "field." If no word anywhere in the message has a match in the emoji keyword list, the keyboard provides no suggestions. Lexical suggestion is implemented using the open-source emoji library emojilib 4. The library provides a *.json file containing 1,502 emojis and their corresponding keywords. For example, the clapping emoji has words "hands," "praise," "applause," "congrats" and "yay." We did not force the frequency of emoji suggestion updates to be the same for both lexical and semantic suggestion mechanisms, as these mechanisms are fundamentally different in nature. For lexical suggestion, the opportune moment for updating the emoji suggestions is straightforward-whenever a keyword has been typed. For semantic suggestion, however, it is unclear when the suggestions should be updated because it is not obvious when the user is finished typing. Thus, our keyboard updates the emoji suggestions after the user finishes typing each word, not just keywords. We also did not force an equal number of emoji suggestions across the two keyboards. Using a machine learning model for semantic suggestion returns a fixed number of emojis (five in DeepMoji), but lexical suggestion can produce a 4 https://github.com/muan/emojilib variable number of emojis. Adding extra emojis when lexical suggestion produces too few emojis would confuse users with unrelated emojis, and conversely, trimming potentially relevant emojis from the semantic suggestions would make for a keyboard unrepresentative of its full potential. Our keyboard logs input statistics related to text and emoji entry: the number of typed characters, the number of deleted characters, the number of emojis manually added from the traditional emoji enumeration interface, and the number of emojis selected from the two suggestion lists. To respect participants' privacy, our keyboard did not log the content of any typed messages. Before having people use our keyboard, we conducted a preliminary crowd-sourced experiment to evaluate the performance of lexical and semantic emoji suggestion systems. Specifically, we were interested in quantifying the mechanisms' human-perceived accuracy. We randomly sampled 50 tweets from the Sentiment140 dataset 5, among which 25 contained positive sentiments and 25 contained negative sentiments. For each tweet, lexical and semantic emoji suggestions were generated and shuffled into a list. The tweets and their corresponding emoji suggestions were added to a single survey task on Amazon Mechanical Turk. Respondents were asked to "choose as many emoji options that made sense to be added at the end of each sentence." If the respondent felt that none of the emoji suggestions were relevant, they were allowed to select "none of the above." Respondents were not told that different suggestion mechanisms were in use. We collected responses from 20 English-speaking Mechanical Turkers. The are shown in Table 2. Overall Relevance refers to the number of selected emojis divided by the total number of emojis shown in the survey. There were 5,000 total emojis shown by the semantic suggestion system (50 tweets × 5 emojis per tweet × 20 people); respondents felt that 52.5% of them aligned with their corresponding tweet. For the lexical suggestion system, 2,520 emojis were shown, of which 21.6% were deemed relevant, less than half of those from the semantic suggestion system. Pick-1 Relevance examines whether any of the predicted emojis were selected for each tweet; in other words, as long as any emoji from a particular suggestion mechanism was deemed relevant for a tweet, the Pick-1 Relevance of that mechanism was 100% for that tweet. Out of 1,000 tweets (50 tweets × 20 users), semantic suggestion provided a relevant emoji 94.6% of the time. Lexical suggestion, on the other hand, only produced a relevant emoji 32.6% of the time. Top-1 Percentage is a head-to-head comparison that captures whether the most commonly selected tweet per emoji came from the lexical or semantic suggestion system. Semantic suggestion was the overwhelming winner, providing the more relevant emoji for 47 of the 50 tweets. The show that Turkers generally perceived the emojis based on semantic content as more relevant than those based on keywords. This verified our hypothesis that the two suggestion mechanisms would be perceived differently. However, this study alone does not demonstrate how the different suggestion mechanisms affect the chatting experience, which leads us to our laboratory experiment. Although in-lab experiments are not generally representative of realistic conditions, they are useful for studying conversations because they allow data to be gathered from both senders and receivers. An in-lab study enabled us to explore how the emoji suggestion mechanisms might affect the two conversational sides differently. If a person uses more emojis because they find it easier to do so, a recipient might react in two ways: they might enjoy the conversation more and reciprocate, or they might enjoy the conversation less due to annoyance, distraction, or confusion. Twenty-six participants (15 females, 11 males) between 18 and 34 years old (M=28.9, SD=4.2) were recruited via emails, word-of-mouth, and convenience sampling in a large university setting. The participants were randomly divided into 13 pairs. The pairs were constructed such that the participants did not know each other and did not meet faceto-face until the end of the study. Each participant was given $8 USD as compensation for the 30-minute study. Participants were provided with Nexus 6P smartphones running Google Android 7.0. Our keyboard was installed on each phone. Wechat 6 was used as the instant message application because Wechat provides a function to export the chat history. We used the chat history to verify the data logged by our mobile keyboard. Participants were instructed to avoid using Wechat's built-in button for emoji entry since it bypassed our keyboard's logging functionality. Participants were told that they would take part in an online chat experiment using our mobile keyboard. They chatted 6 https://www.wechat.com/en/ with another participant for three 10-minute sessions, each of which was assigned to one of three emoji suggestion conditions: no-suggestion, lexical, or semantic. The order of the conditions was fully counterbalanced across participants. The participants were told that they could steer the conversation towards any topic of their choosing but were told that a "recent activity" could be used to start. The participants were also told that the only difference between the sessions would be the keyboard's emoji suggestion , but they were not told anything about the suggestion mechanisms, what they were, or how they worked. Q1. Do you communicate online with your phone (SMS/IM/Email, etc.) a lot? Q2. Do you use emojis in online conversations a lot? Table 3. The questionnaire about online chatting and emoji use behavior. Q1. The chatting experience was interesting. Q2. My attention was focused on the conversation. Q3. I could express my emotion clearly using the keyboard. Q4. I felt constrained in the types of expressions I could make. Q5. I was able to get an impression of my partner. Q6. The chatting experience excites my curiosity. Table 4. The survey questions about the chat experience. Answers were provided via Likert scales ranging from 1 (strongly disagree) to 7 (strongly agree). Q1. I used the emoji suggestion a lot in my typing, and it was useful. Q2. I would like to use this system frequently. Q3. I thought the system was easy to use. Q4. The system did well on proposing relevant emojis. Q5. I like the emoji suggestion system better than the nosuggestion system. Table 5. The usability survey for the suggestion keyboards. Answers were provided via Likert scales ranging from 1 (strongly disagree) to 7 (strongly agree). Before the conversation began, the participants were told to fill out a questionnaire that asked about their online chat and emoji use behaviors (Table 3). After each session, the participants filled out another questionnaire asking about their chat experience (Table 4). This questionnaire probed their engagement (Q1, Q2, and Q6) and perceived expressiveness and clarity (Q3, Q4, and Q5) regarding the chat experience. Both questionnaires were derived from prior work on CMC. When participants used lexical or semantic suggestions in a session, they also completed the usability questionnaire shown in Table 5, which was adapted from the SUS survey. At the end of the 30-minute session, participants were interviewed with two open-ended questions: "How do you like the suggestion keyboards? Do you find they affect you (in negative or positive ways) in online chatting?" and "Do you find any problems with the keyboard suggestion mechanism, or do you have any suggestions?" The study was a single-factor three-level within-subjects design with the suggestion mechanism as the independent variable: no-suggestion, lexical, and semantic. We utilized multiple statistical analyses according to the nature of the dependent variables: character count measures were analyzed using the aligned rank transform procedure; emoji count measures fit a Poisson distribution, and were therefore analyzed with mixed model Poisson regression; Likert-scale responses were treated as ordinal measures and were therefore analyzed with mixed model ordinal logistic regression. Further specifics are given with each analysis in the . In this section, we describe the of the study comparing the three levels of the Suggestion factor: no suggestions, lexical suggestions, and semantic suggestions. During the study, one pair of participants did not conduct what we considered a realistic conversation. In one of their sessions, they sent only nonsensical numbers and capital letters to each other. This participant pair was therefore excluded from our analyses, and another pair was recruited in their place. Thus, our dataset included 12 valid participant pairs with two pairs per Suggestion order due to full counterbalancing (3! conditions = 6 orders). We collected 12×3=36 data logs of valid sessions, together with 72 surveys regarding the chat experience and 48 usability surveys for emoji suggestion. We conducted formal analysis with open coding, in which research team members identified any themes or codes they discovered from the 48 responses to the open-ended questions (Q1-Q2 in Table 3). Among the 24 participants, 22 stated that they always communicate with their phone, while the other two stated that they only used their phone sometimes. Nine participants stated that they always use emojis in online conversations, 14 sometimes, and one seldom. As for how our participants normally enter emojis, 14 participants manually selected emojis from a list, one participant used lexical suggestions from the keyboard, and nine used both methods. The descriptive of the logged data are shown in Table 6. Total Characters is the number of characters excluding emojis sent in the conversation; Total Emojis is the number of emojis used in the conversation, however they might have been inputted; and Selected Emojis is the number of emojis picked from the suggestion list. 7 Note that the no suggestion condition was excluded from this analysis since it did not produce emoji suggestions. =7.76, p<.05), with semantic suggestion ing in more selected emojis than lexical suggestion. 7 This indicates that although the total number of emojis participants used across conditions was similar, participants selected more semanticgenerated emojis than lexical-generated ones. Participants responded to the questionnaires along a 7-point Likert Scale (1=strongly disagree, 7=strongly agree), so the data were analyzed using mixed model ordinal logistic regression. Surprisingly, there were no significant across the different Suggestion levels for any question regarding either the chat experience (Table 4) or usability (Table 5). Based on the analysis of emoji counts in the study, we found that although different suggestion levels ed in similar amounts of inputted emoji, participants tended to pick more from semantic suggestions than from lexical suggestions. One surprising finding was that although the usage of emojis indeed affected the senders' chat experience, the suggestion type did not affect the chat experience significantly. One explanation is that different suggestion mechanisms only affect how the user inputs emojis, rather than what they input. As long as they can input the expected emojis, the chat experience is not affected. Looking at participants' interview answers, we found that participants did notice the difference between the suggestion mechanisms, and provided more positive feedback on semantic suggestions than the other conditions. Five participants mentioned that semantic suggestions were convenient and timesaving. The convenience might come from the relevance of the semantic suggestions. P13 pointed out, "The first one [semantic] is better than the second one [lexical], showing more emotion-related emojis. The second one is related to the word itself and it makes no sense to use the emoji in the conversation." Although P19 did not use many emojis during the study, she stated that "their [emojis'] appearance in suggestion bars makes me feel good." This feedback supports our finding that people chose more emojis from semantic suggestion than lexical suggestion. We also conducted a 15-day field deployment to explore the longitudinal effects of the different emoji suggestion systems. This study focused on the usability of the emoji suggestion systems and on their effects on emoji usage during everyday conversations. Eighteen participants (8 females, 10 males) between 18 and 43 years old (M=24.0, SD=6.4) were recruited via emails, flyers, and word-of-mouth. Inclusion criteria required that participants were able to use English as their primary language and owned a smartphone with Android 6.0 that they used on a daily basis. Those who were in the laboratory study were not allowed to participate in the field deployment due to prior exposure. The 15-day study contained three five-day periods. Participants were compensated $20 USD in the first two periods and $40 for the third, adding to $80 total. The study was conducted as a partial within-subjects design with the suggestion mechanism as the independent variable. All of the participants used the no-suggestion keyboard in the first five-day period as a baseline (however, they could still input emoji from the emoji selection panel). During the second period, half of the participants used the lexical suggestion keyboard while the other half used the semantic suggestion keyboard. Everyone returned to the nosuggestion keyboard during the last period to determine whether they returned to their baseline behavior. In psychology terms, the study compared an ABA condition sequence to an ACA condition sequence. When participants were enrolled, they were asked to fill out the same questionnaire about online chatting and emoji usage as in the laboratory study. Participants were told that they would be using an emoji suggestion system during the field study, but that they were free to use or ignore the suggestions as they pleased. Participants were instructed to use the keyboard whenever they were typing in English and to keep their phone network connected so they could retrieve emoji suggestion . The same usage information was logged as before (Total Characters, Total Emojis, and Selected Emojis). After participants signed the consent form, the keyboard was installed on their phone. The keyboard was configured to participants' personal preferences, including its aesthetic theme and vibration behavior. Participants met with a researcher after each five-day period to have their keyboards reconfigured to another condition and fill out a short questionnaire about the experience (Table 7). After the second period, when emoji suggestions were provided, participants also completed the same usability survey as in the first study (see Table 5). Survey After Period 1 1. Do you find yourself using emojis more or less often than before the study? Why? Survey After Period 2 1. How do you like or dislike the suggestion keyboard? Do you find it affecting you (in negative or positive ways) in online communication? 2. Do you find yourself using emojis more often than before the study? Why? 3. Do you have any comments about the keyboard emoji suggestions? Survey After Period 3 1. What do you think of the current keyboard for this period? 2. Do you find yourself using emojis more or less often than before the study? Why? 3. After the whole period, do you have any comments about the keyboard emoji suggestions? Table 7. The survey questions after each period. The emoji suggestions were offered only during period 2, which is why the questions are different for that period. We collected 54 data logs (18 participants × 3 periods), 18 survey about the usability of emoji suggestions, and 54 open responses analyzed using inductive analysis. As before, Suggestion was the independent variable of three levels: no-suggestion, lexical, and semantic. Among the 18 participants, 14 stated that they always communicate with their phone, three sometimes, and one seldom. Four participants stated that they always use emojis in online conversations, 11 sometimes, and three seldom. As for the participants' typical emoji entry method, 10 participants manually selected emojis from a list, one participant used lexical suggestions from the keyboard, and seven used both methods. The descriptive statistics for Total Characters, Total Emojis and Selected Emojis per day are shown in Unsurprisingly, participants used more emojis with lexical and semantic suggestions than with no suggestions. On average, participants who used lexical suggestions in the second period increased their emoji usage by 31.5% over their baseline, while participants who used semantic suggestions increased their usage by 125.1%. We note that the average usage of daily emoji seems low (fewer than 5 emojis per day). After looking into the data, we found that some participants used over 10 emojis per day, while the other participants used less than one emoji per day. Furthermore, we analyzed the difference in Total Emojis between the different periods by Suggestion using MannWhitney U tests. Results showed that emoji usage increased significantly more with semantic suggestions than with lexical suggestions from the first to second period (p<.001). The change between the first and third periods was not significantly different, indicating that the change in emoji usage was due to the emoji suggestion and not just time. The Likert scale responses from the usability survey during the second period were analyzed using mixed model ordinal logistic regression. No statistically significant differences were found between the semantic and lexical suggestions for any of the questions. The quantitative analysis are similar to the in-lab study: the total emoji inputs were similar between different suggestion levels in period 2, and users chose more semantic suggestions than lexical suggestions. Again, based on the survey , suggestion mechanisms did not influence the online conversation experience significantly., and also provides supporting evidence of why people picked more semantic emojis in our online study. Our goal was to examine the impact of emoji suggestion on online conversations. In particular, we sought to answer two primary questions: How do emoji suggestion systems affect the chat experience? Do lexical and semantic suggestion systems affect daily emoji usage differently? We first conducted an online study to evaluate the performance of the two systems, finding that semantic emoji suggestions were perceived as more relevant than lexical emoji suggestions. We then conducted two experiments, finding that emoji usage had a stronger effect on senders than on receivers, but the suggestion system in use did not affect the overall chat experience. A possible explanation is that the suggestion levels only affect the ease of inputting an emoji. Although participants picked more from the semantic suggestions, they could still manually pick their desired emojis if those emojis were not suggested, leading to similar numbers of total emojis inputted with the different suggestion systems. However, both our in-lab study and our field deployment revealed that the suggestion systems influenced how users reflected on their own experiences. Participants were clearly most excited about semantic suggestions. Even without knowing the details of the different suggestion systems, the participants were pleasantly surprised that the predicted emojis were related to the sentiment of their messages. During the field deployment, participants used more emojis in their daily conversations from semantic suggestions than from lexical suggestions. This finding shows that the semantic suggestions provided more relevant emojis than did the lexical suggestions. Based on feedback from the user studies, we created several design guidelines for future emoji suggestion systems: • Suggestion Diversity. Emoji suggestion systems should suggest various types of emojis, ranging from emojis that portray objects to emojis that portray emotions. Although semantic suggestions were preferred in our study, many participants wanted the system to provide more suggestions than just face emojis. Some participants also appreciated that the lexical suggestion system would sometimes suggest rare emojis. Suggestions from multiple systems could be combined to provide more diverse emojis. Lexical suggestion could provide emojis as the user is typing a sentence, and once the user has finished the sentence, semantic suggestion could provide emojis that reflect the message's overall meaning. Combining the two suggestion schemes could be useful because not all messages contain strong semantic information, and people also use emojis to provide additional information for certain words, such as changing the tone. • Personalization. Beyond providing the most common emoji suggestions, emoji suggestion systems should be aware of the user's personal favorites and usage behaviors. Usage behaviors could be based on categories (e.g., faces, hearts) or the emotions that the user prefers to express. In addition, it would be useful if the suggestion keyboard could recognize the recipient or the usage scenario. For example, a user might want heart emojis when chatting with a family member on a messaging app, but object emojis when composing an email. • Avoiding Intrusion. Emoji suggestion keyboards should only predict emojis when necessary. Some participants only wanted suggestions at the end of messages, as they found the always-on style of semantic suggestions to be distracting. One limitation of our study is that the suggestion frequency of the two emoji systems was not the same. The semantic suggestion system updated with each new word typed, while the lexical suggestion system updated only after each predefined keyword. Thus, participants were exposed to more suggestions in the semantic condition than in the lexical condition. We used our online study to measure the relevance of emoji suggestions independent of frequency. Collecting a similar measure could have been done in our other studies by counting the number of selected emojis and dividing by the total number of emoji suggestions; however, such a metric would neglect many other factors that affect selection rate (e.g., time duration, ordering of emojis). Another limitation is in our keyboard implementation, namely that the existing semantic-level suggestion model we used contains only 64 possible emojis, thus limiting the diversity of possible suggestions. The DeepMoji model could be extended to more emojis, but we chose to stay with the original set to align with the findings from Felbo et al.'s prior work, since there is no available conversation datasets with emojis for fine tuning the model. In this work, we compared two emoji suggestion systems: lexical and semantic. Specifically, we explored whether the suggestion type affected the online chat experience and how people perceive the two suggestion types. Our online crowdsourced study revealed that people perceived semantic suggestions as most relevant. Our laboratory study showed that semantic emoji suggestions were used about 1.5 times more than lexical emoji suggestions. Our longitudinal field deployment showed that semantic suggestions led to an increase in emoji usage and were preferred because of their relevance to emotions. As other research in this area has found, we can conclude that emojis themselves, rather than the type of suggestion system, affects the chat experience most profoundly. Based on our study , we offered design guidelines for emoji suggestion systems. We believe that by incorporating semantic information in emoji suggestion, researchers can provide better experiences in text-based computer-mediated communications.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We built an Android keyboard with both lexical (word-based) and semantic (meaning-based) emoji suggestion capabilities and compared their effects in two different chat studies.

Humans in the loop revise documents to accord with counterfactual labels, resulting resource helps to reduce reliance on spurious associations.


Question: Represent the scientific article to find a short TLDR

Question: Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation. Generative Adversarial Networks ) (GANs) have achieved tremendous success for image generation and received much attention in computer vision. For text generation, however, the performance of GANs is severely limited due to reward sparsity and mode collapse: reward sparsity refers to the difficulty for the generator to receive reward signals when its generated samples can hardly fool the discriminator that is much easier to train; while mode collapse refers to the phenomenon that the generator only learns limited patterns from the real data. As a , both the quality and diversity of generated text samples are limited. To address the above issues, we propose a novel self-adversarial learning (SAL) paradigm for improving adversarial text generation. In contrast to standard GANs (Figure 1(a) ) that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier assessing whether the currently generated sample is better than its previously generated one, as shown in Figure 1 (b). During training, SAL rewards the generator when its currently generated samples are found to be better than its previously generated samples. In the earlier training stage when the quality of generated samples is far below the real data, this self-improvement reward mechanism makes it easier for the generator to receive non-sparse rewards with informative learning signals, effectively alleviating the reward sparsity issue; while in the later training stage, SAL can prevent a sample from keeping receiving high reward as the self-improvement for a popular mode will become more and more difficult, and therefore help the generator avoid collapsing toward the limited patterns of real data. We comprehensively evaluate the proposed self-adversarial learning paradigm in both synthetic data and real data on the text generation benchmark platform . Compared to the previous approaches for adversarial text generation (; ;), our approach shows a substantial improvement in terms of both the quality and the diversity of generated samples as well as better performance stability in adversarial learning. Figure 1: (a) Conventional adversarial learning that uses a binary real/fake classifier as its discriminator; (b): Self-adversarial learning that employs a comparative discriminator to compare the currently generated sample to its previously generated samples for obtaining rewards through self-improvement. Adversarial text generation has drawn much attention in recent years due to its advantages (e.g., sequence-level guidance without the exposure bias issue ) over maximum likelihood estimation (MLE) for natural language generation. It formulates the learning process as a minimax game between a generator G θ parameterized by θ and a discriminator D φ parameterized by φ: the discriminator is trained to distinguish between the samples drawn from the real data distribution p data and the samples generated by the generator; while the generator is trained to generate samples that can "fool" the discriminator. The adversarial learning objective of the generator and the discriminator can be formulated as: where x is a sample from the read data, G θ (z) is a sample generated by the generator with the initialization z that is drawn from the noise distribution p z (e.g., standard normal distribution). While GANs have shown some promising , there are two fundamental issues that impede their progress in text generation: (i) Reward sparsity, which is due to the fact that the discriminator tends to learn much better than the generator and thus easily recognizes generated samples as fake. In such cases, it will be difficult for the generator to receive rewards; (ii) Mode collapse, which arises from the intrinsic nature of GANs and leads the adversarial models to only learn the limited patterns from the real samples. These two issues limit the ability of GANs to generate high-quality and diverse text samples, which have not been well addressed yet. To address the aforementioned issues, we propose a novel self-adversarial learning (SAL) paradigm. Inspired by self-play in reinforcement learning, the core idea of SAL is to reward the generator if its currently generated sample is found to be better than its previously generated ones. Like AlphaGo , the generator in SAL struggles to learn to generate better samples than its previously generated samples for passing the "self-improvement" test by a comparative discriminator, which is a pairwise classifier trained to compare the quality of two samples, as Figure 1 (b) shows. Compared to conventional GANs (Figure 1(a) ), SAL has the following advantages: First, in the earlier training stage when the quality of generated samples is far below the real data, the self-improvement reward mechanism of SAL allows the generator to receive informative learning signals more easily as it makes the assessment of sample quality better adapted to the current capability of the generator, making it less likely to suffer from the issue of reward sparsity; Second, in the later training stage when the generated samples' quality is high, SAL can prevent a sample from keeping receiving high reward as it will become more and more difficult to pass the "self-improvement" test, thus reducing the risk of the generator collapsing towards limited patterns. The self-improvement mechanism and the'tie' option in the comparative discriminator also provides a reasonable baseline which corresponds to cases where newly generated samples are found to be indistinguishable with previous ones, thus improving the training stability. We provide a more detailed qualitative analysis of why the proposed self-adversarial learning paradigm can alleviate these problems in Appendix. As introduced above, the core component for SAL is the comparative discriminator, which is a pairwise classifier comparing the quality of two samples. It learns a total order of sample quality and encodes the inductive bias that one sample is better (>), worse (<), or indistinguishable (≈) in terms of its quality compared to the other. For a (text) sample, the comparative discriminator can offer more informative learning signals than the conventional binary (i.e., real/fake) classification based discriminator because the sample can receive multiple feedbacks from the comparative discriminator by comparing it with multiple other samples. For training the comparative discriminator, we construct pairwise training examples from the real and generated samples, as Figure 2 shows. For a real sample s + and a generated sample s −, we assign the label "better (>)" to the pair (s +, s −) and "worse (<)" to (s −, s +). For two samples both from real data or from the generated samples, we assign the label "indistinguishable (≈)" to such pairs (i.e., (s . For a training set with n real samples and n generated samples, the comparative discrimination can construct 2n 2 pairwise training examples, allowing to enhance the generalization ability of the comparative discriminator. Moreover, to improve the model's ability to distinguish between good generated samples and bad generated samples for self-play learning, we additionally select the samples generated during the later stage of MLE training as pseudo-real samples, and select those generated in the earlier epochs when the generator does not fully converge as fake sentences. We then pair the pseudo-real samples with the fake samples to construct training instances to supervise the model to compare their qualities. In this way, the comparative discriminator is prevented from being taught to always recognize two generated samples as equally bad and assigning zero reward to the generator. As a , it can become more sensitive to the quality difference in a pair of text samples and thus allow the generator to receive rewards more easily. Before we formally introduce the training procedure for SAL, we first define the learning objective for the comparative discriminator D φ and the generator G θ in SAL: In Eq and Eq, M is the set of previous generated samples by the generator, Q(x 1, x 2) ∈ {>, <, ≈} is the true label for the pair (is the probability of the comparative discriminator's prediction being q (q ∈ {>, <, ≈}) for the pair (x 1, x 2). w q is the reward weight for the case q, which is a hyperparameter for SAL. If the generator generates a sample G θ (z) that is better (>) than its previously generated sample x r, it receives a positive reward; if G θ (z) is worse (<) than x r, it receives a negative reward; while if the quality of G θ (z) is classified as similar (≈) to x r, it receives zero credit. Therefore, we have Since L G can only be directly optimized in standard continuous GAN training, we alternatively employ the policy gradient algorithm to train the generator, as previous approaches for adversarial text generation. For SAL, we define the reward for a generated sample x g compared with a reference sample x r which is a previous generated sample by the generator as the weighted reward based on the probability distribution of the prediction by the comparative discriminator: In text generation, the generator G θ obtains the reward only when one sample has been completely generated, which means no intermediate reward is gained. To relieve this problem, following the practice in SeqGAN , we utilize the Monte Carlo rollout method to approximate intermediate rewards by sampling unknown tokens with generated prefix Y 1:t following generator policy G θ till sample completion. Empirically, we found that the Monte Carlo rollout also helps to reduce the variance of the reference sample utilized for comparison. We calculate the expected reward as The objective of the generator is to generate a sequence to maximize its expected final reward. With likelihood ratio , we can formulate the gradient of the objective function for generator G θ as: To improve the training of self-adversarial learning, we borrow ideas from the field of deep reinforcement learning and propose two training techniques to improve self-adversarial training. Scheduled rewarding Similar to the exploitation-exploration trade-off in reinforcement learning , the positive reward assigned for actions generating better samples encourage exploration while the penalty for generating worse samples makes the generator more conservative. Intuitively, in the earlier stage of self-adversarial learning, the generator should explore better policy by receiving higher rewards for relative progress; while in the later stage, the generator should be more conservative by penalizing more for worse samples to avoid performance degradation. We simply decrease w (>) and increase w (<) linearly with training iteration and refer this technique as scheduled rewarding. Memory replay Continuously comparing the generator with its most recent stage may suffer from the correlation between generated samples and reference samples, which makes the training process unstable. Inspired by experience replay , we construct a memory buffer which contains samples generated in the last K training steps. Reference samples are sampled from the memory buffer rather than samples generated in the most recent stage of the generator, which empirically helps stabilize the training process. The training process of SAL is summarized in Algorithm 3. Self-adversarial learning with the proposed comparative discriminator achieves Nash Equilibrium when the generator models the distribution of real samples perfectly. In this case, the comparative discriminator cannot successfully distinguish generated samples from real samples and tends to recognize two samples as "indistinguishable". The reward received by the generator is thus zero and training converges. However, how a non-Bernoulli GAN converges to such an equilibrium is still an open problem and is beyond the scope of this work. Following the experimental settings in previous work (; ; ; ;), we evaluate our approach in both synthetic and real datasets Require: Generator G θ; comparative discriminator D φ; samples of real sentences S+; self-adversarial learning step g; discriminator step k; memory buffer M for the previous generated samples 1: Pretrain G θ using MLE on S+ 2: Generate samples with G θ and store them into M 3: repeat 4: for k steps do 5: Collect a mini-batch of balanced sample pairs (x1,x2) from M ∪S+ 6: Update D φ via Eq 7: end for 8: for g steps do 9: Generate a mini-batch of samples xg ∼ G θ 10: Collect a mini-batch of reference samples xr from M 11: Update G θ via Eq 12: end for 13: Update M with G θ 14: until Convergence based on Texygen , which is a benchmark platform for evaluating adversarial text generation models. Table 1 presents the basic information of the datasets used for evaluation. As SeqGAN , our generator is a single-layer LSTM and our discriminator is almost based on TextCNN except that it concatenates the feature representation of two compared samples and outputs the probability for their comparative relations (i.e., >, <, ≈). We keep the most of the hyperparameters same with the SeqGAN except the hyperparameters introduced by our models (i.e., w (>), w (<), w (≈) ) which are tuned based on the synthetic experiment and kept the same for the real data experiments. We evaluate the adversarial text generation models in terms of both quality and diversity. Following the prior work, in the synthetic dataset, we use the oracle LSTM to evaluate the negative log-likelihood of our generated samples (denoted as NLL oracle) as the metric to reflect quality; for the diversity metric, we use the negative log-likelihood of the synthetic dataset (denoted as NLL gen) evaluated by the generator with the best quality (i.e., the best NLL oracle score) during training. We also use the best NLL oracle + NLL gen obtained during training to evaluate the quality-diversity trade-off. For the real data experiments, we follow the previous work to apply the commonly-used BLEU scores (BLEU(F)) and the perplexity of generated samples evaluated by an open-sourced pretrained language model as quality metrics since NLL oracle cannot be evaluated without an oracle language model. For evaluating diversity, we employ both backward BLEU (BLEU(B)) which evaluates the test data using generated samples as reference and NLL gen as the metrics. To evaluate the generated samples in more aspects, we calculate frechet distance (FD) between generated samples and real data with sentence representation obtained by InferSent which is a pre-trained sentence embedding model. We compare our approach to previous well-known adversarial text generation models including SeqGAN , RankGAN and MaliGAN . Leak-GAN and RelGAN focus on architecture-level modification, which is orthogonal to our work and the proposed self-adversarial learning paradigm can be applied to them as well. We provide of the combination of LeakGAN with SAL in the Appendix. In the following sections, we denote our proposed self-adversarial learning approach as SAL. Since adversarial training is very sensitive to random initialization and suffers from high variance, we conduct five individual runs with different random seeds for each model and report the mean and the standard deviation of the obtained . Table 2 shows the in the synthetic dataset. We can observe that SAL largely outperforms the previous GANs in all metrics in both cases of sequence length 20 and 40. Although its performance in NLL gen is worse than MLE as MLE directly optimizes the metric, it yields better quality-diversity trade-off than MLE training, which has not been achieved by the previous GANs, which is shown by the fact that the NLL oracle +NLL gen for SAL is lower than that yielded by MLE, while other GANs have the same sum score with MLE, indicating that they fail to improve the quality-diversity trade-off after pretraining. This demonstrates SAL's advantage in alleviating the mode collapse problem. We also find that the training of SAL is more stable compared with other text GANs. In addition, we find SAL learns faster and better than the other GANs by comparing their performance curves of NLL oracle during training in Figure 3, which is consistent with our intuition that the selfimprovement reward mechanism in SAL can alleviate the reward sparsity issue in the earlier training stage and help the generator learn better. The for COCO image caption dataset are presented in Table 3 and the performance curve of the perplexity during training is shown in Figure 4. As in the synthetic data, we observe that our SAL consistently yields better in all the metrics with stable performance (i.e., low variance) compared to the previous GANs. According to Table 18 and Figure 4, SeqGAN and our SAL can improve MLE in the quality metrics (i.e., BLEU (F) and Perplexity) while MaliGAN and RankGAN perform comparably to MLE. However, in the WMT NEWS dataset where text samples tend to be longer, we observe something different from Table 4: all the previous GANs fail to improve MLE. This is because long text generation makes the discrepancy between generated samples and real samples very large even after MLE pre-training. As a , previous GANs fail to stably enhance the sample quality due to the reward sparsity issue. In contrast, our SAL consistently performs well Table 4: Performance comparison of different models in the EMNLP2017 WMT news generation task. Metrics from top to bottom represent respectively the generation quality, the generation diversity, and the divergence between real and generated data. For all the BLEU metrics, the higher, the better. For NLL gen and FD, the lower, the better. and largely improves the quality metrics over MLE. In addition, we observe that the diversity of samples generated by SAL is much better than previous GANs and only marginally worse than MLE, indicating SAL is helpful in reducing the risk of mode collapse. In addition to the automatic metrics, we also conduct human evaluation for the generated samples. As previous work , we randomly sample 20 sentences from each model and pool them with anonymizing the models' identity. We invite 20 graduate students with good English proficiency to score each sentence on a scale of 1-5 in terms of quality. According to Table 5, our SAL is consistently well rated in the human evaluation and outperforms all the baselines in both COCO and WMT NEWS datasets. We perform the Wilcoxon Rank Sum Test with the human evaluation and find that samples generated by baseline models can be distinguished from samples generated by SAL with p < 0.01. Details of human evaluation procedure and samples generated by compared methods in two real-world datasets are presented in the Appendix. To better understand SAL, we perform multiple ablation tests in both the synthetic and the real data. We employ NLL oracle + NLL gen score with sequence length 20 as the evaluation metric for the synthetic data, denoted as NLL. For the real data, we use the perplexity of generated samples trained with COCO dataset as the evaluation metric. We compare SAL with the following reduced models: • CAL: Replacing the comparison between the generated samples (i.e., self-play) to the comparison between the real and generated samples. • w/o comparative: Using the binary discrimination scores of other generated samples as baseline for the policy gradient algorithm, which can be considered as a combination of the self-critic training with RL-based text GANs. • w/o "≈": Replace the three-class comparative discriminator with a binary comparative discriminator by removing the "≈" class. • w/o scheduled rewarding and w/o memory replay The of the ablation tests are shown in Table 6. By observing the improvement by SAL over CAL, we confirm the importance of the self-play paradigm in SAL. It is notable that the proposed comparative discriminator alone (i.e., CAL) can yield good performance, demonstrating the effectiveness of learning by comparison. When replacing the comparative discriminator with the naive combination of self-critic baseline with text GANs, the performance largely decreases because the reward sparsity issue will be intensified when subtracting two already sparse rewards, this motivates the proposed pairwise comparative discriminator which makes self-comparison possible. In addition, we find that the "≈" option plays a critical role in improving the , without which the performance degrades significantly because it makes the task less trivial and provides a baseline for the policy gradient algorithm. Moreover, the training techniques (i.e., scheduled rewarding and memory replay) borrowed from deep reinforcement learning are also shown useful in improving the but not so important as the core components (e.g., self-play and the comparative discriminator). , LeakGAN , and RelGAN ) have been proposed for text generation as adversarial training has received increasing attention in recent years. Typically, they address the non-differentiable issue by making continuous approximation or reinforcement learning. These approaches introduce several different architectures and optimization objectives of both the generator and the discriminator for adversarial text generation. Among the previous studies for adversarial text generation, the most related work to ours is RankGAN which proposes a ranker to replace the conventional binary classifier as its discriminator for allowing the discrimination process to involve richer information. Another work whose idea is similar to ours is the relativistic discriminator (RGAN). It compares binary scores assigned to generated samples and real samples by subtraction as the learning signal to implicitly represent the inductive bias that half of the samples received by the discriminator is fake. In contrast, our comparative discriminator directly encodes this inductive bias and assesses generated sentences by comparison with a pairwise classifier, which provides more informative learning signals than subtraction in RGAN and normalized feature similarity in RankGAN . We present a self-adversarial learning (SAL) paradigm for adversarial text generation. SAL rewards the generator when its comparative discriminator finds the generator becomes better than before. Through the self-improvement reward mechanism, the problem of reward sparsity and mode collapse can be alleviated and training of text GANs are more stable, which in a better performance in the text generation benchmarks in terms of both quality, diversity, and lower variance. In the future, we plan to generalize our approach to other domains and modals to explore the potential of SAL for adversarial learning. Generated samples are presented in the Appendix together with other details, including human evaluation details and qualitative analysis of the proposed SAL. A GENERATED SAMPLES We present sentences generated by our proposed model and compared models to provide qualitative evaluation of different adversarial text generation models. From the presented generated samples, we can observe that samples generated by MLE training are less realistic compared with other samples. SeqGAN yield slightly better sample quality but the loss of diversity is observable even within randomly sampled 15 sentences. Adversarial training with proposed comparator, when trained by comparing with real samples, yield better quality but still lack of diversity. Finally, with the proposed self-adversarial learning paradigm, both quality and diversity of generated samples are improved. A.1 GENERATED SAMPLES IN IMAGE COCO DATASET Table 7: Samples generated by SAL in Image COCO dataset a picture of a person's umbrella in a cell phone. a man stands in a green field. a young boy riding a truck. a man on a motorcycle is flying on a grassy field. a girl on a motorcycle parked on a city street. a motorcycle parked in a city street. a group of bikers riding bikes on a city street. a kitchen with a cat on the hood and a street. a bathroom containing a toilet and a sink. a young woman in a kitchen with a smiley face. a jet plane on the side of a street. a dish is sitting on a sidewalk next to a baby giraffe. a dog on a large green bike parked outside of the motor bike. a person on a kawasaki bike on a race track. a commercial aircraft is parked in front of a kitchen. Table 8: Samples generated by CAL in Image COCO dataset a man is on a towel on a table outside of a real kitchen. a group of lambs at a tall building. a young boy riding a truck. a man on a motorcycle is flying on a grassy field. a man with a computer desk next to a white car. a cat is on the walls of a cat. a plane on a runway with a plane. an elegant, dilapidated plane are standing in front of a parking bag. the woman is riding a bike on their way. a man wearing an old bathroom with a banana. a plane is taking off from the ground. a man holding a man in front of herself. a woman is walking across the road. a kitchen with an island in green tiles. a clean kitchen with two small appliances. Table 9: Samples generated by SeqGAN in Image COCO dataset a large image of a herd of racing train. man and woman on horse. a plane on a runway with a plane. a man preparing a table with wood lid. a view, tiled floors and a man prepares food. a man wearing an old bathroom with a banana. a man is is with a camera. two people are parked on a street. a white and white black kitten eating on a table. a toilet is lit on the walls. a kitchen is taking off from a window. a man is wearing glasses wearing scarf. a kitchen with graffiti hanging off from an open plain. two women playing with the orange. a kitchen with an island in a clear glass. Table 10: Samples generated by MLE in Image COCO dataset a jet airplane flies flying through front from an airplane. a furry tub and overhead pot. a man in a kitchen filled with dark lights green side,.. a cross baby field dressed making cardboard a bathroom with a small tub and oven. a man above a bathroom with an oven room. a jet airliner flying through the sky. a kitchen with a dishwasher, and plenty of pots, pans. a person holding onto two red era arena sits on the street. a bathroom with a toilet and a bath tub. a cat perched on the phone and a baseball cap. the view of the street filled with really parked at the gates on the road. a large hairy dog on a high bike with a cake. a man is riding a white back bench. a narrow bed and white spotted dark tiled walls. A.2 GENERATED SAMPLES IN EMNLP2017 WMT DATASET Table 11: Samples generated by SAL in EMNLP2017 WMT dataset it' s likely to be egyptian and many of the canadian refugees, but for a decade. the ministry spokesperson also said it now significant connected to the mountain. it is the time they can more competitive, where we have another $ 99. 100 per cent, and completely on the alternative, and that' s being affected. we expect $ 200 and 0. 3 percent for all you form other, and, which then well, it' s done. so we wouldn' t feel very large in the game, but you fail to fund, and and the paper that' s like its start. other countries made a playoff cut with pages by mrs. trump' s eighth consecutive season as a president. Table 12: Samples generated by CAL in EMNLP2017 WMT dataset i didn' t put relatively quiet, we have,' his work right in the particular heat rate, take steps traditionally clean. why the u. s. then the table is our cabinet to do getting an vital company for the correct review. those had trained for that, but no thin percentage of the nhs about being warned about the palestinian election before obama is not connected in israel. in course, voters -obama said: " torture is the outcome, the most powerful tradepopularity is happening in it as a success. " in 2012, it is nice to remain -no trump actor established this night -scoring three films. we kind of not listen to knowing my most one, only, for a really good vote, and where things fun, you know. Table 13: Samples generated by SeqGAN in EMNLP2017 WMT dataset his missed 4, 000 the first 95 really 69 -year -olds. but just things, you want to thank it as my playing side has begun meeting with " and " the score had to train up, so he was tied for 11 years. and when he got back doing fresh ties with his election, he will now step in january, back. when you' t know if i saw her task to find himself more responsibility ago. his hold over -up to a nine hike in 2015, 13 percent of recently under suspects dead day, 24, and to the city. " i look up on by the city' s vehicle on the day in a meeting in november. Table 14: Samples generated by MLE in EMNLP2017 WMT dataset you know that that is great for our ability to make thinking about how you know and you? when it' s a real thing possible, is if you the first time in a time here and get. u. s, now government spending at the second half of four years, a country where the law will join the region to leave japan in germany. deputy president, the issue of government and geneva probe threats and not -backed trump, but well -changing violence for their islamic state militants were innocent people. he suggested in a presidential primary source and comment on its size following protests conducted by 18, some in 2012 will be looked at tech energy hub. " it' s growing heavy hard, " mr. romney said, he says matters that can' t again become the asian player. In this section, we present several qualitative case study examples to illustrate why comparative discrimination and self-adversarial learning helps mitigate the problem of reward sparsity and mode collapse. We extract a typical sentence generated during the initial stage of adversarial learning: "a man holding a suitcase holding a phones." We see that this sentence is of limited quality and is easily recognized by binary discriminator in SeqGAN with high confidence, this makes the credit received by the generator very sparse and makes training difficult. Comparative adversarial learning (CAL) where we use the comparative discriminator to assess the quality of this sample by comparing it with a real sample helps as comparative discrimination have three catagories, which is less trivial. The improvement is not so much as the discrepancy of generated samples and real samples is fairly large. However, with proposed self-adversarial learning paradigm, the comparative discriminator assesses this sentence by comparing it with a previous generated sentence which is also of poor quality. The self-improvement is easier to be recognized by the comparative discriminator and makes this sample get good rewards. As the comparative discriminator has to learn a total order of sample quality which is more challenging than standard binary discrimination, the chance of the comparative discriminator to be over-trained is reduced, which makes the model easier to achieve self-improvement, thus help to alleviate the reward sparsity problem. We also extract a sentence generated in the late stages which is fairly good and fools the binary discriminator: "a woman sitting on a bench on a park." In standard adversarial text generation models, a sentence like this would keep receiving large rewards and in mode collapse. In self-adversarial learning paradigm, this sentence is not much better than other sentences generated by the generator itself, so its reward is limited, which reduces the risk of mode collapse. For the ablated model variants, SAL w/o self-play and w/o comparative discriminator is trained with the following algorithms. Specifically, the difference between SAL and CAL is that the reference sample which is compared with the currently generated sample is replaced by a real sample instead of a previously generated one. For the variant without the comparative discriminator, we employ a binary discriminator trained in the same way with the vanilla GAN, as for the reward of generating x g, we first sample a previously generated sample x r as reference and calculate the reward as that the self-BLEU scores are always 1 when evaluating all the models. This problem is also discussed in the openreview of the RelGAN paper. Based on this consideration, we decide to employ backward-BLEU which is introduced in and can also evaluate the diversity of generated samples. The forward and backward BLEU resemble precision and recall of generated samples with respect to the test data, which measures the generation quality and the generation diversity respectively. For BLEU metric, as there is no sentence level alignment for unconditional generation, BLEU is evaluated by using the entire test set treated as a single reference, which contains 10000 sentences. We then generate the same number of sentences as the prediction, and then compute n-gram overlap between the reference and the prediction. We did not apply brevity penalty following previous works. But we found the number of tokens generated are roughly the same across different compared models. We briefly explain why NLL gen is able to measure the diversity of the generator: NLL gen measures the negative log-likelihood of the synthetic dataset evaluated by the generator. Intuitively, if the generator is diverse and captures more patterns in the synthetic dataset, the NLL gen score will be lower. In contrast, if the generator suffers from severe mode collapse and is of low diversity, the NLL gen will be higher as the generator fails to cover the diverse patterns in the training data. As for the metric: NLL gen +NLL oracle, our motivation is that NLL oracle measures the best quality of the generator throughout training, while NLL gen measures the best diversity attained by the generator during training. However, the best quality and diversity are generally not achieved at the same time as GAN-training generally sacrifices the diversity for better quality. Therefore, we report NLL gen +NLL oracle which can measure the quality-diversity trade-off, as the previous work demonstrated, as an additional reference. We follow most of the hyperparameters used in the benchmark platform. Specifically, we choose batch size to be 64, dropout keep prob to be 0.75, l2 regularization to be 0.2. We pretrain all model for 120 epochs and fine-tune them until convergence. The proposed self-adversarial learning paradigm introduces the relative weights for credit assignment when a generated sample is found to be better, indistinguishable or worse compared with another sample generated by the generator itself. We tuned it based on the performance in synthetic experiment and find w 0: w 2 = 1: −0.1 to be a good choice for the initial weights 1 and fixed w 1 to be 0. We empirically find that the performance of SAL is not very sensitive to the choice of reward weights as long as the absolute value of w 1 is larger enough than w 2, which guarantees the training stability. Following the human evaluation setting in , The text quality evaluation is based on grammatical correctness and meaningfulness (i.e. if a sentence makes sense), text formatting problems (e.g., capitalization, punctuation, spelling errors, extra spaces between words and punctuations) are ignored. Detailed criteria is provided as follows: It has some small grammatical errors and mostly makes sense. For example, "two women is in a cafe look outside." 3-Fair It has major grammatical errors but the whole still conveys some meanings. For example, "a man riding on on motor scooter and window." 2-Poor It has severe grammatical errors and the whole doesn't make sense, but some parts are meaningful. For example, "a blue bike on on on a dirt bike." 1-Unacceptable It is basically a random collection of words. For example, "a motorcycle close close on it and." C.5 ADDITIONAL ON CAL AND LEAKGAN In this section, we present the performance comparison on SAL vs CAL (comparative adversarial learning, which uses comparative discriminator but does not train the model with self-play.) and the application of SAL on LeakGAN. We find that SAL significantly outperforms CAL on all dataset. In addition, we find that the proposed self-adversarial learning paradigm can also be applied on other text GAN architectures, e.g. LeakGAN, and help improve its performance.

Represent the short summary for finding the corresponding scientific article (long)

We propose a self-adversarial learning (SAL) paradigm which improves the generator in a self-play fashion for improving GANs' performance in text generation.

We design an adversarial training method to Bayesian neural networks, showing a much stronger defense to white-box adversarial attacks