Question: Represent the scientific article to find a short TLDR

Question: Multi-agent collaboration is required by numerous real-world problems. Although distributed setting is usually adopted by practical systems, local range communication and information aggregation still matter in fulfilling complex tasks. For multi-agent reinforcement learning, many previous studies have been dedicated to design an effective communication architecture. However, existing models usually suffer from an ossified communication structure, e.g., most of them predefine a particular communication mode by specifying a fixed time frequency and spatial scope for agents to communicate regardless of necessity. Such design is incapable of dealing with multi-agent scenarios that are capricious and complicated, especially when only partial information is available. Motivated by this, we argue that the solution is to build a spontaneous and self-organizing communication (SSoC) learning scheme. By treating the communication behaviour as an explicit action, SSoC learns to organize communication in an effective and efficient way. Particularly, it enables each agent to spontaneously decide when and who to send messages based on its observed states. In this way, a dynamic inter-agent communication channel is established in an online and self-organizing manner. The agents also learn how to adaptively aggregate the received messages and its own hidden states to execute actions. Various experiments have been conducted to demonstrate that SSoC really learns intelligent message passing among agents located far apart. With such agile communications, we observe that effective collaboration tactics emerge which have not been mastered by the compared baselines. Many real-world applications involve participation of multiple agents, for example, multi-robot control BID12, network packet delivery BID20 and autonomous vehicles planning BID0, etc.. Learning such systems is ideally required to be autonomous (e.g., using reinforcement learning). Recently, with the rise of deep learning, deep reinforcement learning (RL) has demonstrated many exciting in several challenging scenarios e.g. robotic manipulation BID3, visual navigation BID22 BID10, as well as the well-known application in game playing BID13 etc.. However, unlike its success in solving single-agent tasks, deep RL still faces many challenges in solving multi-agent learning scenarios. Modeling multiple agents has two extreme solutions: one is treating all agents as an unity to apply a single centralized framework, the other is modelling the agents as completely independent learners. Studies following the former design are often known as "centralized approach", for example BID18 BID14 etc. The obvious advantage of this class of approaches is a good guarantee of optimality since it is equivalent to the single agent Markov decision process (MDP) essentially. However, it is usually unfeasible to assume a global controller that knows everything about the environment in practice. The other class of methods can be marked as "independent multi-agent reinforcement learning". These approaches assumes a totally independent setting in which the agents treat all others as a part of the observed environment. BID2 has pointed out that such a setup will suffer from the problem of non-stationarity, which renders it hard to learn an optimal joint policy. In essence, there are three key factors that determine a communication. That is when, where and how the participants initiate the communication. Most of existing approaches, including the abovementioned Meanfield and Commnet, try to predefine each ingredient and thus lead to an inflexible communication architecture. Recently, VAIN BID4 and ATOC BID6 incorporate attentional communication for collaborative multi-agent reinforcement learning. Compared with Meanfield and Commnet, VAIN and ATOC have made one step further towards more flexible communication. However, the step is still limited. Take ATOC as an example, although it learns a dynamic attention to diversify agent messages, the message flow is only limited to the local range. This is unfavorable for learning complex and long range communications. The communication time is also specified manually (every ten steps). Hence it is requisite to find a new method that allows more flexible communication on both learnable time and scopes. In this regard, we propose a new solution with learnable spontaneous communication behaviours and self-organizing message flow among agents. The proposed architecture is named as "Spontaneous and Self-Organizing Communication" (SSoC) network. The key to such a spontaneous communication lies in the design that the communication is treated as an action to be learned in a reinforcement manner. The corresponding action is called "Speak". Each agent is eligible to take such an action based on its current observation. Once an agent decides to "Speak", it sends a message to partners within the communication scope. In the next step, agents receiving this message will decide whether to pass the message forward to more distant agents or keep silence. This is exactly how SSoC distinguishes itself from existing approaches. Instead of predestining when and who will participate in the communication, SSoC agents start communication only when necessary and stop transferring received messages if they are useless. A self-organizing communication policy is learned via maximizing the total collaborative reward. The communication process of SSoC is depicted in Fig.1. It shows an example of the message flow among four communicating agents. Specifically, agent 3 sends a message to ask for help for remote partners. Due to agent 3's communication range, the message can be seen only by agent 1. Then agent 1 decides to transfer the collected message to its neighbors. Finally agent 2 and agent 4 read the messages from agent 3. These two agents are directly unreachable from agent 3. In this way, each agent learns to send or transfer messages spontaneously and finally form a communication route. Compared with the communication channels predefined in previous works, the communication here is dynamically changing according to real needs of the participating agents. Hence the communication manner forms a self-organizing mechanism. We instantiate SSoC with a policy network with four functional units as shown in FIG0. Besides the agent's original action, an extra "Speak" action is output based on the current observation and hidden states. Here we simply design "Speak" as a binary {0, 1} output. Hence it works as a "switch" to control whether to send or transfer a message. The "Speak" action determines when and who to communicate in a fully spontaneous manner. A communication structure will naturally emerge after several steps of message propagation. Here in our SSoC method, the "Speak" policy is learned by a reward-driven reinforcement learning algorithm. The assumption is that a better message propagation strategy should also lead to a higher accumulated reward. We evaluate SSoC on several representative benchmarks. As we have observed, the learned policy does demonstrate novel clear message propagation patterns which enable complex collaborative strategies, for example, remote partners can be requested to help the current agent to get over hard times. We also show the high efficiency of communication by visualizing a heat map showing how often the agents "speak". The communication turns out to be much sparser than existing predefined communication channels which produce excessive messages. With such emerged collaborations enabled by SSoC's intelligent communication manner, it is also expected to see clear performance gains compared with existing methods on the tested tasks. Recently, several studies have concentrated on learning multiple agent communication for deep RL networks. Among them, one class of work tries to build pairwise structure for multi-agent communication. BID1 and BID8 are among the first to propose learnable communications via back-propagation between individual deep Q-networks. However, due to their motivating tasks, both works rely on a peer-to-peer (P2P) communication strategy and usually apply to only a limited number of agents. VAIN BID4 also builds a pair-wise communication structure whose interaction weights are learned. BID15 proposes a pair-wise MARL approach for multi-object tracking task. In such models, the essential state-action space of multiple agents grows geometrically with the number of agents, and so is the communication load. Hence the pair-wise models suffer from scale issues. Another branch of study attempts to establish a global communication channel for all agents. For example, BID14 proposes a bidirectional communication channel among all agents to facilitate effective communication. BID18 applies a centralized method with zero-order optimization to control all agents. BID7 uses a master-slave architecture to control the communication of local agents by a master. Commnet BID17 builds a broadcasting communication channel among all agents and mean hidden states are directly used as messages. Such architectures are practical communication ways but still limited since 1) such a predefined structure forces every agent to send messages at every step. This will produce lots of redundant information; 2) They all use a single net to model all agents, which makes it difficult to extend to large-scale tasks. To address the large-scale multi-agent problem in MARL, BID19 proposes Meanfield algorithm. It models a local communication between each agent and an approximating "mean-agent" of all its neighbors. This encourages information sharing within a local range, however, its capability is still limited since the average of action-value function may not be rich enough to induce complex collaborative policies for complicated tasks. Several recent studies exploit a "centralized/communicated training + decentralized/independent execution" mode. For example, in COMA BID5, a global critic was proposed, which could potentially work at a centralized level, however since critics are basically value networks, they do not provide explicit policy guidance. BID11 also presents an adaptation of actor-critic method which uses centralized action-value function that takes as input the states and inferred actions of all agents. However, this mainly fosters a better training of each independent model instead of establishing an explicit communication and information processing module at testing time which is essential for effective collaborations In order to learn more effective communication, BID6 uses a recurrent attention model to help dynamically perform communication and have better local communication. However, its message flow is only limited to the local range. This is unfavorable for learning complex and long range communications. The communication time is also specified manually (every ten steps). Compared with their work, SSoC learns both when and who to communicate with a spontaneous communication action. It also allows long range communication via multi-step message propagation. These characteristics are critical for learning better collaborative policies. Here we introduce the details of the proposed SSoC network. We assume a distributed partially observable MARL environment for SSoC. As illustrated by Fig.1, each agent can only see a neighboring circular area. The goal is to achieve a certain winning state via taking a sequence of actions. At each time t, an agent takes an action a t based on its own observed state s t as well as information sent by partner agents within a local range around it. In this paper we call the shared information m i t as "messages". The messages flowing among agents determine which agents are sharing information with others at a certain time step. Unlike previous methods which usually predestine a static communication structure, SSoC adopts a more flexible option. The key is to enable each agent to learn an extra "Speak" action to determine whether to send the message to neighboring agents. Hence only those agents "Speak" currently are able to be "heard" by its partners within its communication range. Then these partner agents are able to exploit the received messages to determine their own actions. They are also eligible to pass the received messages forward to its neighbors if they also take a "Speak" action. And their own "thought" can also be merged into the output message. Like ordinary actions, "Speak" action also takes one time step. A communication path will form naturally among all agents through multiple steps of communication. SSoC manages to establish a "global" communication channel in such a self-organizing manner. Unlike previous communication architectures, this "global" communication channel of SSoC is also applicable for a distributed MARL scenario. The "Speak" action is generated by an independent output of the network, which is parallel to the original action output of each agent. We design "Speak" action as a binary signal to control whether a message will be passed. Hence the action space of "Speak" contains 2 actions: "1" represents "Speak", "0" stands for "keep silent". This binary "Speak" signal will be multiplied with the output message of the agent. If the signal is "1", the output message is preserved and will be accessible by the neighbor agents' LWU module. Otherwise, it is hidden from other agents by a 0 signal. Since the "Speak" action is determined by the agent itself based on its own observation, the subsequent communication is started in a spontaneous way. It may happen anytime, anywhere which only depends on the agents' own policy. It is neither predetermined nor manually designed with human intuitions. Therefore, we name such kind of communication as "self-organizing". Here we re-check the three key factors of multi-agent communication mentioned in section 1. Obviously, such a "self-organizing" communication of SSoC is capable of learning when and who to communicate simultaneously. As for "how to communicate", each agent is able to combine the received message with its own "thought" to output actions. Furthermore, since SSoC implicitly builds a "global" communication channel based on multi-step of "Speak" actions, the communication is not only limited to the local area. Therefore, SSoC's self-organizing communication really addresses the key challenges of multi-agent communication in a new and more flexible way. The whole structure of SSoC is displayed as FIG0. The network mainly consists of two streams. The top "agent stream" of FIG0 shows the original policy network of each agent. By taking the input observation, a policy network will generate a hidden state "thought" h t. Then the "thought" h t will be merged via an ACU (action composition unit) module with messages m i t from the bottom "message stream" to obtain the action a t. And an extra binary "Speak" action will also be generated from h t to decide whether to send the agent's message to others. The message stream works as a message receiver & sender for the agent. At each time step, it aggregates messages from other agents within the communication range via a LWU (learnable weighting unit). Only messages from agents who "Speak" at last time step will get collected by LWU. To incorporate the agent's own "thought" and send it to other agents, the aggregated messages m i t will be combined with h t through a MCU (message composition unit) module as the output message m o t. Finally, the binary "Speak" action from agent stream will be applied to determine the sending of message m o t to other agents. FIG0 gives detailed implementation of the four functional units of SSoC. Specifically, the policy network consists of 3 fully-connected layers The LWU module works as a self-weighted aggregator of the collected messages {m o1 t, m o2 t, m o3 t, . . .} from neighbor agents. For each input message, it will predict a weight which will be multiplied with the original signal before a sum operation with other messages. MCU merges the agent's own "thought" and input message as a new message that will be spread to other agents. It adopts a learnable gating mechanism which enables the agent to decide the proportion of its own "thought" and other agents' information in its output message. Like MCU, ACU also combines the input message with its own "thought". However, its goal is to output the agent's action. It is implemented like a basic LSTM cell without temporal recurrence. The agent's own action will be updated by a regular policy gradient with baseline. And the "Speak" action takes an additional "speak" policy gradient as equation 1: DISPLAYFORM0 where T is the total time step of the episode. These two gradients will be fed to SSoC network respectively for back-propagation (as shown by the green arrows in FIG0). Note that the output messages will be re-used as the input of other agents in next time step. Hence we create a buffer to store the messages m o t at each time step. During training, messages will be taken out by each agent from this buffer for both feed-forward sampling and backward updates. The detailed training procedure is given in the appendix. Experiments are conducted on multi-camera intelligent surveillance task which is firstly proposed by this paper and the large-scale battle task (64 vs. 64) from the MAgent benchmark BID21. Both tasks assume distributed collaborative MARL environments. Agents only have local observations and take actions independently. A shared reward will be given to all agents for model training. For comparison, we select two state-of-the-art methods BID17 as well as the independent multi-agent policy gradient algorithm (denoted as Independent) as our baselines. In real life there are many occasions that need multiple agents cooperation to achieve their common goal. One interesting case is to make a set of intelligent cameras track target people in airport or railway station by actively changing their pose. Each camera is manipulated by an agent. Nearby agents can share their information with each other. Inspired by this scenario, we design a simulation task which we name as multi-camera intelligent surveillance task (as illustrated in FIG1 . In this task, each target person (simplified as the small red balls) has his/her own walking route. They will appear in the entrance area randomly at the beginning and walk to the destination. There are three self-controlling cameras (the blue sector represents the camera's current visual field) trying to actively capture the moving target balls as long as possible. The camera can choose to sway its angle left, right (by 15 degree) or keep static. Each camera can share its own "thought" only with its neighbor camera. Hence this is a distributed MARL setting. The general objective of the cameras is to capture the moving balls as long as possible by changing their angles cooperatively. We formulate the multi-camera intelligent surveillance task as a multiagent reinforcement learning problem:• Overall objective: tracking the target red balls as long as it can during the whole episode;• State: the sector visual field's position and the ball's position (given only if it is within the visual field, otherwise is empty);• Action: turning left by 15 degree, turning right by 15 degree or stay static;• Reward: if the camera changes its angle and captures the target ball, it receives a positive reward; if the ball is in the camera's possible visual field but the camera doesn't capture it, the agent receives a negative reward; Otherwise, it receives a zero reward. (The details of reward definition is given in appendix.)Next we will show how our SSoC works in this environment. We argue that the camera agent knows nothing about ball's information, it needs to learn the useful information based its local observation by itself. In our algorithm, the agent will adjust and optimize it's policy with training and learn to dynamically send necessary information to its local partners. As shown in FIG2, when the ball is captured by the camera, it chooses to speak "1" meaning that it has collected the ball's information and send the integrated message to the next camera to help the next camera learning better. When the ball is outside of all the cameras' field, the cameras choose to speak "0" meaning that there is no useful and necessary message to be delivered between cameras. We compare SSoC on the multi-camera intelligent surveillance task with Meanfield, Commnet and Independent MARL. In our environment setting, each episode contains 30 time steps And we train every algorithm for 5000 episodes in total. We show the of SSoC with baseline algorithms table 1. We find that the agents learn a smart strategy that once the agent captures the ball, it can capture the ball all the time until the ball moves outside its visual field. In addition, once an agent captures a ball, its neighbor agent will search around by turning back and forth to capture the coming ball earlier. This collaborative strategy is enabled by useful messages passing among the agents. When necessary, the agent learns to speak "1" meaning that it decides to transfer the useful message to next agent. When there is no effective message, the agent chooses to be silent. For better illustration, we visualize SSoC's learned policy in the video against other algorithms. The video is in demo. We recommend readers to have a look at the video for better understanding of the policy learned by SSoC. The large-scale battle task is one MARL scenario included in the MAgent environment BID21. In the task our algorithm controls a group of agents to eliminate the opponents. Each agent can move by one cell or attack one nearby enemy at one time step. This task is challenging due to a large number of participating agents (here 64 in our experiments). The difficulty is even larger considering the environment's distributed setting and agents' partial visibility. We adopt a self-play training for all the methods following BID21. All methods are trained for 2000 episodes. Each episode contains 400 steps. Learning rate is set to 0.0001 for all. SSoC outperforms the compared baselines on mean rewards and mean kill (average number of killed enemies in an episode) with a clear margin as shown by table 2. We also let SSoC play against Commnet, Meanfield and Independent in this task. The are shown in 6 (b). As we can see, SSoC obtains a higher win rate compared with the baselines. The demonstrate SSoC is an competitive architecture for distributed MARL tasks. To verify if it is SSoC's spontaneous and self-organizing communication that brings such a big improvement, we draw the heatmap of communication in Fig.5. The heat value here is computed as an accumulation of the "Speak" signal (1 for "Speak" and 0 for silence) of each cell in recent 5 frames. In this way, we discover several cases with meaningful message flow which help our agents win the battle. One such case is shown in Fig.5. Here we display both the heatmap and corresponding situation in 10 th, 15 th and 20 th frame here to show the learned agent policies and message flow. In this case, frontier agents (marked as group a) encounter enemies first and needs support of the agents of group b. Hence they start communication by taking "Speak" actions at 10 th frame. The messages sent by these agents transfer to group b through several steps. At 15 th frame, group b agents receive the message and "Speak" to gather nearby agents to move right and help agents on the right. At 20 th step, group b agents begin to confront enemies and join the attack. After a while, two groups of agents join force together to eliminate most of the enemies. This example shows that the learned self-organizing communication does help our agents to learn a better collaborative policy. Since the proposed scheme enables multi-step message transferring, an agent's message can be spread to agents on a much further area. Hence such a long-range reinforcement policy can be learned in this case. In addition, we can see for most agents, "Speak" action is not taken. This shows that agents only start a communication when necessary, instead of keeping sending messages all the time like in BID17 BID14. This phenomenon shows the high efficiency of SSoC communication. Table 2: mean rewards on larget-scale battle task In this paper, we propose a SSoC network for MARL tasks. Unlike previous methods which often assume a predestined communication structure, the SSoC agent learns when to start a communication or transfer its received message via a novel "Speak" action. Similar to the agent's original action, this "Speak" can also be learned in a reinforcement manner. With such a spontaneous communication action, SSoC is able to establish a dynamic self-organizing communication structure according to the current state. Experiments have been performed to demonstrate better collaborative policies and improved on communication efficiency brought by such a design. In future work, we will continue to enhance the learning of "Speak" action e.g. encoding a temporal abstraction to make the communication flow more stable or develop a specific reward for this "Speak" action.

This paper proposes a spontaneous and self-organizing communication (SSoC) learning scheme for multi-agent RL tasks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We study the BERT language representation model and the sequence generation model with BERT encoder for multi-label text classification task. We experiment with both models and explore their special qualities for this setting. We also introduce and examine experimentally a mixed model, which is an ensemble of multi-label BERT and sequence generating BERT models. Our experiments demonstrated that BERT-based models and the mixed model, in particular, outperform current baselines in several metrics achieving state-of-the-art on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts. Multi-label text classification (MLTC) is an important natural language processing task with many applications, such as document categorization, automatic text annotation, protein function prediction , intent detection in dialogue systems, and tickets tagging in client support systems . In this task, text samples are assigned to multiple labels from a finite label set. In recent years, it became clear that deep learning approaches can go a long way toward solving text classification tasks. However, most of the widely used approaches in MLTC tend to neglect correlation between labels. One of the promising yet fairly less studied methods to tackle this problem is using sequence-to-sequence modeling. In this approach, a model treats an input text as a sequence of tokens and predict labels in a sequential way taking into account previously predicted labels. used Seq2Seq architecture with GRU encoder and attention-based GRU decoder, achieving an improvement over a standard GRU model on several datasets and metrics. Yang et al. (2018b) continued this idea by introducing Sequence Generation Model (SGM) consisting of BiLSTM-based encoder and LSTM decoder coupled with additive attention mechanism. In this paper, we argue that the encoder part of SGM can be successfully replaced with a heavy language representation model such as BERT . We propose Sequence Generating BERT model (BERT+SGM) and a mixed model which is an ensemble of vanilla BERT and BERT+SGM models. We show that BERT+SGM model achieves decent after less than a half of an epoch of training, while the standard BERT model needs to be trained for 5-6 epochs just to achieve the same accuracy and several dozens epochs more to converge. On public datasets, we obtain 0.4%, 0.8%, and 1.6% average improvement in miF 1, maF 1, and accuracy respectively in comparison with BERT. On datasets with hierarchically structured classes, we achieve 2.8% and 1.5% average improvement in maF 1 and accuracy. Our main contributions are as follows: 1. We present the of BERT as an encoder in the sequence-to-sequence framework for MLTC datasets with and without a given hierarchical tree structure over classes. 2. We introduce and examine experimentally a novel mixed model for MLTC. 3. We fine-tune the vanilla BERT model to perform multi-label text classification. To the best of our knowledge, this is the first work to experiment with BERT and explore its particular properties for the multi-label setting and hierarchical text classification. 4. We demonstrate state-of-the-art on three well-studied MLTC datasets with English texts and two private Yandex Taxi datasets with Russian texts. Let us consider a set D = {(x n, y n)} N n=1 ⊆ X × Y consisting of N samples that are assumed to be identically and independently distributed following an unknown distribution P (X, Y). Multiclass classification task aims to learn a function that maps inputs to the elements of a label set L = {1, 2, . . ., L}, i.e. Y = L. In multi-label classification, the aim is to learn a function that maps inputs to the subsets of L, i.e. Y = 2 L. In text classification tasks, X is a space of natural language texts. A standard pipeline in deep learning is to use a base model that converts a raw text to its fixedsize vector representation and then pass it to a classification algorithm. Typical architectures for base models include different types of recurrent neural networks (;, convolutional neural networks , hierarchical attention networks, and other more sophisticated approaches. These models consider each instance x as a sequence of tokens x = [w 1, w 2, . . ., w T]. Each token w i is then mapped to a vector representation u i ∈ R H thus forming an embedding matrix U T ×H which can be initialized with pre-trained word embeddings . Moreover, recent works show that it is possible to pre-train entire language representation models on large corpora of texts in a selfsupervised way. Newly introduced models providing context-dependent text embeddings, such as ELMo , ULMFiT , OpenAI GPT , and BERT significantly improved previous state-of-the-art on various NLP tasks. Among the most recent works, XLNet and RoBERTa models improve these further after overcoming some limitations of original BERT. A novel approach to take account of dependencies between labels is using Seq2Seq modeling. In this framework that first appeared in the neural machine translation field , we generally have source input X and target output Y in the form of sequences. We also assume there is a hidden dependence between X and Y, which can be captured by probabilistic model P (Y|X, θ). Therefore, the problem consists of three parts: modeling the distribution P (Y|X, θ), learning the parameters θ, and performing the inference stage where we need to findŶ = arg Y max P (Y|X, θ). have shown that after introducing a total order relation on the set of classes L, the MLTC problem can be treated as sequence-to-sequence task with Y being the ordered set of relevant labels {l 1, l 2, . . ., l M} ⊆ L of an instance X = [w 1, w 2, . . ., w T]. The primary approach to model sequences is decomposing the joint probability P (Y|X, θ) into M separate conditional probabilities. Traditionally, the left-to-right (L2R) order decomposition is: demonstrated that the label ordering in effects on the model accuracy, and the order with descending label frequencies in a decent performance on image datasets. Alternatively, if an additional prior knowledge about the relationship between classes is provided in the form of a tree hierarchy, the labels can also be sorted in topological order with a depth-first search performed on the hierarchical tree. argued that both orderings work similarly well on text classification datasets. A given hierarchical structure over labels forms a particular case of text classification task known as hierarchical text classification (HTC). Such an underlying structure over the set of labels can help to discover similar classes and transfer knowledge between them improving the accuracy of the model for the labels with only a few training examples . Most of the researchers' efforts to study HTC were dedicated to computer vision applications (; ; ;), but many of these studies potentially can be or have already been adapted to the field of natural language texts. Among the most recent works, proposed a Graph-based CNN architecture with a hierarchical regularizer, and argued that mixing an output from a global classifier and the outputs from all layers of a local classifier can be beneficial to learn hierarchical dependencies. It was also shown that reinforcement learning models with special award functions can be applied to learn non-trivial losses (a; . BERT (Bidirectional Encoder Representations from Transformers) is a recently proposed language representation model for obtaining text embeddings. BERT was pre-trained on unlabelled texts for masked word prediction and next sentence prediction tasks, providing deep bidirectional representations. For classification tasks, a special token [CLS] is put to the beginning of the text and the output vector of the token [CLS] is designed to correspond to the final text embedding. The pretrained BERT model has proven to be very useful for transfer learning in multi-class and pairwise text classification. Fine-tuning the model followed by one additional feedforward layer and softmax activation function was shown to be enough for providing state-of-the-art on a downstream task . For examining BERT on the multi-label setting, we change activation function after the last layer to sigmoid so that for each label we predict their probabilities independently. The loss to be optimized will be adjusted accordingly from cross-entropy loss to binary cross-entropy loss. In sequence generation model (b), the authors use BiLSTM as an encoder with pre-trained word embeddings of dimension d = 512. For a raw text x = [w 1, w 2, . . ., w T] each word w i is mapped to its embedding u i ∈ R d, and contextual word representations are computed as follows: After that, the decoder's zeroth hidden state is initialized as We propose to use the outputs of the last transformer block in BERT model as vector representations of words and the embedding of the token [CLS] produced by BERT as the initial hidden state of the decoder. We also use a simple dot-product attention mechanism which in our setting showed similar performance as additive attention, but ed in less number of parameters to learn. The process we follow to calculate decoder's hidden states α t and the attention scores α t is described in Algorithm 1 and illustrated in Figure 1. The weight matrices It is also worth mentioning that we do not freeze BERT parameters so that they can also be fine-tuned in the training process. In order to maximize the total likelihood of the produced sequence, we train the final model to minimize the cross-entropy objective loss for a given x and ground-truth labels {l * In the inference stage, we can compute the objective 3 replacing ground-truth labels with predicted labels. To produce the final sequence of labels, we perform a beam search following the work to find candidate sequences that have the minimal objective scores among the paths ending with the <EOS> token. In further experiments, we mainly test standard BERT and sequence generating BERT models. From our experimental that will be demonstrated later on, we concluded that BERT and BERT+SGM may each have their advantages and drawbacks on different datasets. Therefore, to make the models alleviate each other's weaknesses, it might be reasonable to combine them. Our error analysis on a number of examples has shown that in some cases, BERT can predict excess labels while BERT+SGM tends to be more restrained, which suggests that the two approaches can potentially complement each other well. Another argument in favor of using a hybrid method is that in contrast to the multi-label BERT model, BERT+SGM exploits the information about the underlying structure of labels. in their work propose HMCN model in which they suggest to jointly optimize both local (hierarchical) and global classifiers and combine their final probability predictions as a weighted average. Inspired by this idea, we propose to use a mixed model which is an ensemble of multi-label BERT and sequence generating BERT models. A main challenge in creating a mixed model is that the outputs of the two models are quite different. Typically, we do not have access to a probability distribution over the labels in classic Seq2Seq framework. We suggest to tackle this problem by computing the probability distributions produced by the decoder at each stage and then perform element-wise max-pooling operation on them following the idea of the recent paper . We should emphasize that using these probabilities to produce final label sets will not necessarily in the same predictions as the original BERT + SGM model. However, in our experiments, we found that the probability distributions obtained in that way are quite meaningful and with proper prob- Table 1: Summary of the datasets. N is the number of documents, L is the number of labels, W denotes the average number of words per sample ± SD, and C denotes the average number of labels per sample ± SD. ability threshold (around 0.4-0.45 for the considered datasets) can yield predictions with accuracy comparable to the accuracy of BERT+SGM model's predictions from the inference stage. After obtaining probability distributions of both models, we can compute their weighed average to create the final probability distribution vector, as follows: This probability vector is then used to make final predictions of labels with 0.5 probability threshold. The value of α ∈ is a trade-off parameter that is optimized on validation set. The final procedure is presented in Algorithm 2. pBERT ← BERT(x) [y1, y2, . . ., yn] ← BERT+SGM(x) for l ∈ {1, 2, . . ., L} do p (l) BERT+SGM ← max{y 1l, y 2l, . . ., y nl} pmixed ← αpBERT+SGM + (1 − α)pBERT L pred ← {l | p (l) mixed We train and evaluate all the models on three public datasets with English texts and two private datasets with Russian texts. The summary of the datasets' statistics is provided in the Table 1. Preprocessing of the datasets included lower casing the texts and removing punctuation. For the baseline TextCNN and SGM models, we used the same preprocessing techniques as in (b). Reuters Corpus Volume I (RCV1-v2) is a collection of manually categorized 804 410 news stories (after dropping four empty samples from the testing set). There are 103 categories organized in a tree hierarchy, and each text sample is assigned to labels from one or multiple paths in the tree. Since there was practically no difference between topological sorting order and order by frequency in multi-path case, we chose to sort the labels from the most common ones to the rarest ones. The training/testing split for this dataset is originally 23,149 in the training set and 781,261 in the testing set . While this training/testing split is still used in modern research works (;, in some other works authors have (implicitly) shifted towards using reverse training/testing split , and several other recent research works (; a ;b) started using 802,414 samples for the training set and 1,000 samples for the validation and testing sets. This change of the split might be reasonable due to the inadequate original proportion of the sets in modern realities, yet it makes it difficult to perform an apple-to-apple comparison of different models without their reimplementation. To avoid confusion, we decided to be consistent with the original training/testing split. We also used 10% of the training data for validation. Reuters-21578 is one of the most commonly used MLTC benchmark datasets with 10,787 articles from Reuters newswire collected in 1987 and tagged with 90 labels. We use the standard ApteMod split of the dataset following the work . Arxiv Academic Paper Dataset (AAPD) is a recently collected dataset (b) consisting of abstracts of 55,840 research papers from arXiv.org. Each paper belongs to one or several academic subjects, and the task is to predict those subjects for a paper based on its abstract. The number of categories is 54. We refer the reader to Appendix B for visualization of multi-label BERT embeddings for some of the labels from this dataset. Riders Tickets from Yandex Taxi Client Support (Y.Taxi Riders) is a private dataset obtained in Yandex Taxi client support system consisting of 174,590 tickets from riders. Initially, the dataset was labeled by Yandex Taxi reviewers with one tag per each ticket sample with an estimated accuracy of labeling around 75-78%. However, using additional information about a tree hierarchical structure over labels, we substituted each label with the corresponding label set with all the parent classes lying in the path between the root node and the label node. After this procedure, we ended up with 426 labels. Since in this task there is only one path in the tree to be predicted, we will explore a natural topological label ordering for this dataset. An example of a subtree of the tree hierarchy is provided in Figure 2. Drivers Tickets from Yandex Taxi Client Support (Y.Taxi Drivers) is also a private dataset obtained in Yandex Taxi drivers support system which has similar properties with the Y.Taxi Riders dataset. In the drivers' version, there are 163,633 tickets labeled with 374 tags. We implemented all the experiments in PyTorch 1.0 and ran the computations on a GeForce GTX 1080Ti GPU. Our implementation is relied on pytorch-transformers library 1. In the experiments, we used the base-uncased versions of BERT for English texts and the base-casedmultilingual version for Russian texts. Models of both versions output 768-dimensional hidden representation vector. We set batch size to 16. For optimization, we used Adam optimizer with β 1 = 0.9, β 2 = 0.99 and learning rate 2 · 10 −5. For the multi-label BERT, we also used the same scheduling of the learning rate as in the original work by. Reuters Table 2: Results on the five considered datasets. Metrics are marked in bold if they contain the highest metrics for the dataset in their ±SD interval. Following previous research works , we used hamming accuracy, set accuracy, micro-averaged f 1, and macro-averaged f 1 to evaluate the performance of the models. To be specific, the former two metrics can be computed as ACC(y,ŷ) = 1(y =ŷ) and HA(1(y j =ŷ j) and are designed to determine the accuracy of the predicted sets as whole. The latter ones are label-based metrics and can be calculated as follows: where tp j, f n j, and f p j denote the number of true positive, false positive and false negative predictions for the label j, respectively. We use a classic convolutional neural network TextCNN as a baseline for our experiments. We implemented a two-layer CNN with each layer followed by max pooling and two feedforward fully-connected layers followed by dropout and batch normalization at the end. Our second baseline model is Sequence Generation Model SGM (b), for which we reused the implementation of the authors 2. For the sake of comparison, we also provide the of HMCN and HiLAP (Mao et al.) models for hierarchical text classification on RCV1-v2 dataset adopted from the work (Mao et al.). For Reuters-21578 dataset, we also included the of the EncDec model from the original paper on sequence-to-sequence approach to MLTC. We present the of the suggested models and baselines on the five considered datasets in Table 2. First, we can see that both BERT and BERT+SGM show favorable on multi-label classification datasets mostly outperforming other baselines by a significant margin. On RCV1-v2 dataset, it is clear that the BERT-based models perform the best in micro-F 1 metrics. The methods dealing with the class structure (tree hierarchy in HMCN and HiLAP, label frequency in BERT+SGM) also have the highest macro-F 1 score. In some cases, BERT performs better than the sequence-to-sequence version, which is especially evident on the Reuters-21578 dataset. Since BERT+SGM has more learnable parameters, a possible reason might be a fewer number of samples provided on the dataset. However, sometimes BERT+SGM might be a more preferable option: on RCV1-v2 dataset the macro-F 1 metrics of BERT + SGM is much larger while other metrics are still comparable with the BERT's . Also, for both Yandex Taxi datasets on the Russian language, we can see that the hamming accuracy and the set accuracy of the BERT+SGM model is higher compared to other models. On Y.Taxi Riders there is also an improvement in terms of macro-F 1 metrics. In most cases, better performance can be achieved after mixing BERT and BERT+SGM. On public datasets, we see 0.4%, 0.8%, and 1.6% average improvement in miF 1, maF 1, and accuracy respectively in comparison with BERT. On datasets with tree hierarchy over classes, we observe 2.8% and 1.5% average improvement in maF 1 and accuracy. Metrics of interest for the mixed model depending on α on RCV1-v2 validation set are shown in Figure 4. Visualization of feature importance for BERT and sequence generating BERT models is provided in Appendix A. In our experiments, we also found that BERT for multi-label text classification tasks takes far more epochs to converge compared to 3-4 epochs needed for multi-class datasets . For AAPD, we performed 20 epochs of training; for RCV1-v2 and Reuters-21578 -around 30 epochs; for Russian datasets -45-50 epochs. BERT + SGM achieves decent accuracy much faster than multi-label BERT and converges after 8-12 epochs. The behavior of performance of both models on the validation set of Reuters-21578 during the training process is shown in Figure 3. Another finding of our experiments is that the beam size in the inference stage does not appear to influence much on the performance. We obtained optimal with the beam size in the range from 5 to 9. However, a greedy approach with the beam size 1 still gives similar with less than 1.5% difference in the metrics. A possible explanation for this might be that, while in neural machine translation (NMT) the word ordering in the output sequence matters a lot and there might be confusing options, label set generation task is much simpler and we do not have any problems with ordering. Also, due to a quite limited'vocabulary' size |L|, we may not have as many options here to perform a beam search as in NMT or another natural sequence generation task. In this research work, we examine BERT and sequence generating BERT on the multi-label setting. We experiment with both models and explore their particular properties for this task. We also introduce and examine experimentally a mixed model which is an ensemble of vanilla BERT and sequence-to-sequence BERT models. Our experimental studies showed that BERT-based models and the mixed model, in particular, outperform current baselines by several metrics achieving state-of-the-art on three well-studied multi-label classification datasets with English texts and two private Yandex Taxi datasets with Russian texts. We established that multi-label BERT typically needs several dozens of epochs to converge, unlike to BERT+SGM model which demonstrates decent just after a few hundreds of iterations (less than a half of an epoch). A natural question arises as to whether the success of the mixed model is the of two models having different views on text features. To have a rough idea of how the networks make their prediction, we visualized the word importance scores for each model using the leave-one-out method in Figure 5. It can be seen from this example that BERT+SGM seems to be slightly more selective in terms of features to which it pays attention. Also, in this particular case, the predictions of sequence generating BERT are more accurate. BERT multi-label Figure 5: Visualization of feature importance for multi-label BERT and BERT+SGM models trained on AAPD and applied to BERT paper abstract (cs. LG -machine learning; cs. CL -computation & linguistics; cs. NE -neural and evolutionary computing). We extracted and projected to 2D-plane the label embeddings obtained from the fully connected classification layer of multi-label BERT fine-tuned on AAPD dataset. Visualization of some labels is shown in Figure 6. From this plot, we can see some clusters of labels that are close in terms of word. Figure 6: Projection of label embeddings obtained from the fully connected classification layer of multi-label BERT fine-tuned on AAPD dataset.

Represent the short summary for finding the corresponding scientific article (long)

On using BERT as an encoder for sequential prediction of labels in multi-label text classification task

represent the tldr for finding the corresponding scientific article

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the scientific article to find a short TLDR

Question: Click Through Rate (CTR) prediction is a critical task in industrial applications, especially for online social and commerce applications. It is challenging to find a proper way to automatically discover the effective cross features in CTR tasks. We propose a novel model for CTR tasks, called Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). Instead of learning the cross features directly, DeepEnFM adopts the Transformer encoder as a backbone to align the feature embeddings with the clues of other fields. The embeddings generated from encoder are beneficial for the further feature interactions. Particularly, DeepEnFM utilizes a bilinear approach to generate different similarity functions with respect to different field pairs. Furthermore, the max-pooling method makes DeepEnFM feasible to capture both the supplementary and suppressing information among different attention heads. Our model is validated on the Criteo and Avazu datasets, and achieves state-of-art performance. This paper studies the problem of predicting the Click Through Rate (CTR), which is an essential task in industrial applications, such as online advertising, and e-commerce. To be exact, the advertisements of cost-per-click (CPC) advertising system are normally ranked by the eCPM (effective cost per mille), which is computed as the prodcut of bid price and CTR (click-through rate). To predict CTR precisely, feature representation is an important step in extracting the good, interpretable patterns from training data. For example, the co-occurrence of "Valentine's Day", "chocolate" and "male" can be viewed as one meaningful indicator/feature for the recommendation. Such handcrafted feature type is predominant in CTR prediction , until the renaissance of Deep Neural Networks (DNNs). Recently, a more effective manner, i.e., representation learning has been investigated in CTR prediction with some works (; ; ; ;), which implicitly or explicitly learn the embeddings of high-order feature extractions among neurons or input elements by the expressive power of DNNs or FM. Despite their noticeable performance improvement, DNNs and explicit high order feature-based methods (; ;) seek better feature interactions merely based on the naive feature embeddings. Few efforts have been made in addressing the task of holistically understanding and learning representations of inputs. This leads to many practical problems, such as "polysemy" in the learned feature embeddings existed in previous works. For example, the input feature'chocolate' is much closer to the'snack' than'gift' in normal cases, while we believe'chocolate' should be better paired with'gift' if given the occurrence input as "Valentine's Day". This is one common polysemy problem in CTR prediction. Towards fully understanding the inputs, we re-introduce to CTR, the idea of Transformer encoder , which is oriented in Natural Language Processing (NLP). Such an encoder can efficiently accumulate and extract patterns from contextual word embeddings in NLP, and thus potentially would be very useful in holistically representation learning in CTR. Critically, the Transformer encoder has seldom been applied to CTR prediction with the only one exception arxiv paper AutoInt , which, however, simply implements the multi-head selfattention (MHSA) mechanism of encoders, to directly extract high-order feature interactions. We argue that the output of MHSA/encoder should be still considered as first-order embedding influenced by the other fields, rather than a high-order interaction feature. To this end, our main idea is to apply the encoder to learn a context-aware feature embedding, which contains the clues from the content of other features. Thus the "polysemy" problem can be solved naturally, and the second-order interaction of such features can represent more meaning. Contrast to AutoInt , which feeds the output of encoder directly to the prediction layer or a DNN, our work not only improves the encoder to be more suitable for CTR task, but also feeds the encoder output to FM, since both our encoder and FM are based on vector-wise learning mechanism. And we adopt DNN to learn the bit-wise high-order feature interactions in a parallel way, which avoids interweaving the vector-wise and bit-wise interactions in a stacked way. Formally, we propose a novel framework -Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). DeepEnFM focuses on generating better contextual aligned vectors for FM and uses DNN as a bit-wise information supplement. The architecture adopting both Deep and FM part is inspired by DeepFM . The encoder is endowed with bilinear attention and max-pooling power. First, we observed that unlike the random order of words in a sentence, the features in a transaction are in a fixed order of fields. For example, the fields of features are arranged in an order of {Gender, Age, Price ...}. When the features are embedded in dense vectors, the first and second vectors in a transaction always represent the field "Gender" and "Age". To make use of this advantage, we add a bilinear mechanism to the Transformer encoder. We use bilinear functions to replace the simple dot product in attention. In this way, feature similarity of different field pairs is modeled with different functions. The embedding size in CTR tasks is usually around 10, which allows the application of bilinear functions without unbearable computing complexity. Second, the original multi-head outputs are merged by concatenation, which considers the outputs are complementary to each other. We argue that there are also suppressing information between different heads. We apply a max-pooling merge mechanism to extract both complementary and suppressing information from the multi-head outputs. Experimental on Criteo and Avazu datasets have demonstrated the efficacy of our proposed model. CTR prediction is a critical task in Recommendation System , which aims to predict the probability of a user click behavior based on the given item. The key challenge of the CTR is to automatically learn combinatory/cross features (e.g., a 2-way feature: "Valentine's Day" and "chocolate", a 3-way feature: "Valentine's Day", "chocolate" and "male") from the inputs. Traditional learning methods, such as Logistic Regression (LR) , learn weights of processed features in supervised way for prediction. To gain a better performance, the manually designed combinatory features are fed into LR model, which is laborious and unable to cover the numerous combination of features. To learn the combinatory features automatically in a explicit way, Rendle proposes Factorization Machine model, which treats the weight of a feature pair as the dot of two latent vectors. Moreover, HOFM and CIN are proposed to capture the interaction of arbitrary order in a explicit manner. However, the primary/first-order field embeddings are utilized without considering the contextual information, which leads to the'polysemy' problem in field embedding unsolved. Deep Neural Networks (DNN) have made remarkable progresses in CV and NLP tasks by its extraordinary representation learning power. Several DNN based CTR models are developed on the basis of primary field embeddings;;; ) or product layers with a plain DNN. One of the major drawbacks of the plain DNN is that it tackles the field embeddings as a large flat layer, which means there are only interactions between neurons. Thus, DNN can be utilized as a bit-wise interaction feature learner to assist the vector-wise learner. Recently, Attention mechanism, which has been widely used in Machine Translation and Image Caption tasks, has been adopted in CTR models . Specially, Transformer encoder, empowered with multi-head self-attention, is demonstrated to be capable of generating context-aware word embeddings in the work BERT . Therefore, it seems promising to use Transformer encoder to solve the'polysemy' problem in CTR. Thus, we propose our model DeepEnFM to employ the encoder for context-aware field embedding. A related work to our model is AutoInt . However, AutoInt only utilizes the multi-head self-attention of encoder for high-order feature extraction rather than for field embedding. Task Formulation. The CTR task is to predict the click behavior from a record of multiple fields. Assume we have a training record as {x, y}, where x represents the fields of instance and y ∈ {0, 1} represents the click behavior. The instance fields x may contain both categorical fields and numerical fields. We denote y = 1 when the click behavior is'click'; otherwise, y = 0. Given a unseen test record x *, the goal is then to learn a mapping function y * = Ψ (x *) using all available training information and predict the probability of user click behavior y *, where the y * ∈ represents the probability of click behavior. Overview. Our framework DeepEnFM is illustrated in Fig. 1(a), which is composed of five components: the embedding layer, encoder, FM, DNN and prediction layer. The workflow is depicted in Alg. 1. Firstly, the embedding layer projects the input fields into a low dimensional space. Then the encoder adaptively updates the embedding with respect to the clues from other fields through the multi-head self-attention module (MHSA). Next, FM calculates a score which integrates the explicit interactions between encoder outputs. Simultaneously, the original from the embedding layer are fed into DNN to learn implicit feature interactions at bit-wise level. Finally, we apply the prediction layer to map the intermediate outputs of the FM and DNN into the final score with sigmoid function. Embedding. The embedding layer converts the instance input x from a high dimensional space to a low dimensional, dense vector. As shown in Fig. 1(b), we denote the record input fields as x = [x 1, x 2, . . ., x M], where x i can be either a one-hot vector (categorical field) or a scalar (numerical field). We assume that the field embedding size is d, and the output of embedding layer can be arranged as a matrix E = [e 1 ; e 2 ; . . . ; e M]. For a categorical field, the field value is used to look up the corresponding embedding vector as the token index in word embedding. For a numerical field, the field index is used to look up the related embedding vector, and the field value impacts the embedding as a factor. Encoder. The encoder is a stack of L encoder layers with residual connection, which is designed to align the field embedding accumulatively according to the other field contents. The output of L-th layer can be denoted as E (L). As shown in Fig. 1(c), the encoder layer contains two modules: a multi-head self-attention module and a position-wise feed-forward module. At the end of each module, there is a residual connection followed by the layer normalization. We will further explain these two modules in Sec. 3.2. FM. FM is leveraged to explicitly capture the first order and second order feature interactions as a routine in practice. In our model, FM calculates the score based on the encoder outputs E (L), which are the context-aware field embeddings. DNN. The DNN aims to capture the implicit high-order interactions between different fields in the bit-wise level. The output of embedding layer is flattened to a large vector. Then the large vector is fed into a multiple-layer feed-forward neural network to extract features. We do not use the output of encoder as the input of DNN, because both the encoder and FM learn at vector-wise level, while DNN aims at bit-wise level. Sharing encoder input may jeopardize the performance of encoder. Prediction. The prediction layer is a full connection layer, using a simple logistic regression function based on the outputs of FM and DNN to predict the final . Multi-head Self-attention Module (MHSA). The procedure of MHSA is described in Alg. 2 and Fig. 2(a). Suppose the number of heads is h for each head of MHSA, it has the following steps. The input vector set E is mapping into Query Q, Key K and Value The similarity matrix S is calculated by each pair of query q and key k vectors. The output is calculated as a weighted sum of value vectors v according to S. Finally, the outputs of different attention heads are merged to the final output. As shown in Alg. 2, the similarity and merge functions are the key parts of MHSA. The design principle of similarity and merge function mainly depends on the generalization, efficiency and architecture requirements. We will discuss the different implements between MHSA in Transformer and DeepEnFM encoder. As for MHSA in Transformer encoder, we firstly introduce the similarity function in Transformer encoder as shown in Eq. 4. The similarity function is a scaled dot function, which computes the similarity between different feature pairs equally. It merges the output vectors by a concatenation and a linear transformation with W O ∈ R Hdv×d. for i, j ∈ {1, . . ., M} do 4: 5: 6: end for 9: end for MHSA in DeepEnFM Encoder. To be clear, we argue that the similarity function between different field pairs could be personalized since the field order is fixed and the field embedding size is very low in CTR. Thus we customize the similarity function for each field pair query q (h) i and key k (h) j by performing a bilinear function with a specified weight matrix W (h) (i,j), as shown in Fig. 2(b). Second, we believe that different heads extract features from different viewpoints, so the contain both complementary and suppressing information to each other. Instead of concatenating all of the head outputs, we keep V ∈ R m×d with the same dimensions of E, and use a max-pooling to get the salient output O. Further, we take the output O of MHSA as the residual as shown in Fig. 1 (c). Position-wise Feed-forward Module. The position-wise/field-wise feed-forward module is a one-hidden-layer neural network that firstly maps the field embedding to a higher dimensional embedding space, then back to the original dimensional space. The encoder, FM and DNN in our DeepEnFM can be combined in parallel or stacked way, we explore different architectures for ablation study. We consider that the encoder can serve as a em- bedding layer for the downstream task, as the BERT in NLP tasks . Specially, we stack the encoder and FM together as the left branch as shown in Fig. 1(a), since both of them learn at vector-wise level. Three different architectures are described in Fig. 3, which will be further discussed in ablation study. We conduct experiments on two datasets. Criteo dataset. It contains 45 million click records. Each record includes 13 dense features and 26 categorical ones. For a fair comparison with the other methods, we follow the standard split here: 80%, 10% and 10% of the data as the train, validation and test data respectively. Avazu dataset. It contains 40 million click records, 23 fields totally. We also randomly select 80%, 10% and 10% data for train, validation and test respectively. The categorical features with the occurrence more than 10 and 5 times are preserved for Criteo and Avazu respectively, others are set to'Unknown' as the data processing in AutoInt . Evaluation. The validation data is used to search for hyper-parameter setting. When the hyperparameters determined, both the the train and validation data are used for training, and the performance is evaluated on the test data. The experiment is evaluated on AUC (Area Under ROC) and Logloss (Cross Entropy). Setting. We implement our approach by TensorFlow. We choose Adam as optimizer and the learning rate is 0.001. We set the L2 regularization λ as 0.0001 and dropout rate as 0.5. The batch size, embedding size, and DNN layer size is set to 2048, 10 and 400 respectively. We use Xavier initialization for dense layers and random normal initialization for bilinear attention weights. The DeepEnFM model is trained on V100 for 1 hours per training epoch. Competitors. We compare our model against several state-of-the-art methods: LR . LR simply learns weights of first-order features. FM . FM learns a latent vector for each feature and models feature interactions by dot products of these vectors. This model learns both first and second-order features. DNN. DNN takes embedded dense vectors as its input and learns high-order features in an implicit way. IPNN . IPNN feeds the dense embeddings to a inner product layer firstly. Then the output is fed to a DNN. OPNN . OPNN replaces IPNN's inner product layer with an outer product layer. This replacement reduce the complexity of the model.PNN* . PNN* use the product layer to concatenate the inner and outer product. DeepFM . DeepFM introduces an embedding layer to get dense latent vectors for FM. Then these dense vectors are also fed to a DNN in parallel. AutoInt . AutoInt feeds the output of its embedding layer to a interacting layer. Then MHSA based interacting layer can learn high-order features, and a sigmoid output layer or DNN is applied after the interacting layer. DeepEnFM. Our model employs a encoder-enhanced FM with a DNN. We evaluate the CTR with the standard settings on Criteo and Avazu dataset, as shown in Tab. 1. We highlight the following observations. Our model achieves the best performance of both AUC and LogLoss metrics on Criteo. Specifically, on Criteo our DeepEnFM outperforms the AutoInt by 0.56%, DeepFM by 1.49% on AUC respectively. And DeepEnFM also reduces 0.0067 and 0.0140 on the Logloss comparing to AutoInt and DeepFM respectively. These strongly demonstrate the efficacy of our bilinear encoder. On Avazu dataset, our model achieves the second place in AUC and the first place in Logloss. While, DeepEnFM has still outperformed AutoInt and DeepFM on AUC and Logloss, which indicates the efficacy of our bilinear attention and encoder. The improvement of DeepFM over FM shows the contribution of implicit high-order information in DNN part. The gap between FM and LR reflects that the explicit second order interaction information is important for the CTR task. In summary, all of the implicit high-order information, explicit second order interaction information, and attentional information are critical for the CTR task. We conduct extensive ablation study of DeepEnFM as follows: The in the second part of Tab. 2 shows that'Encoder + Deep' is better than'Encoder + FM'. The gap to the full model demonstrates the effect of each component. Encoder Position. To evaluate the influence of encoder position, we build several variants with the encoder in different position as shown in Fig. 3.'Encoder for DNN': we feed the encoder output to DNN, instead to FM;'Encoder for Both': we feed the encoder output to both the DNN and FM;'Encoder in Parallel': the original embedding output is fed into DNN, FM and encoder simultaneously. Our DeepEnFM can be viewed as'Encoder for FM', which uses the encoder to align the embedding for better explicit feature interaction learning. As shown in the third part of Tab. 2, the show that our DeepEnFM model'Encoder for FM' achieves the best , which indicates that the encoder is more compatible with FM, since both the encoder and FM extract features at vector-wise level. The'Encoder in Parallel' model achieves the second place, since they can learn in an independent way. The'Encoder for Both' model is less effective than DeepEnFM, which indicates that serving simultaneously for both the vector-wise and bit-wise learning modules may harm the power of encoder. Depth of encoder layers. We conduct experiments to evaluate the impact of the encoder layers in DeepEnFM, compared with the depth of MHSA layers in AutoInt. Table 3 shows that the performance of DeepEnFM has been improved as the depth of layers grows, and the gain of of the first encoder layer is larger than the second layer. Particularly, we can see the improvement of an encoder layer is larger than a MHSA layer. Number of Heads. We implement DeepEnFM with the head number of 1, 2, 5. Table 4 shows that the model achieves the best when the number of heads is 1 for DeepEnFM. This is reasonable, the query and key vector dimension reaches the maximum when the number of heads is 1, since the query vector dimension in MHSA is calculated as d h. With more query and key dimension, the bilinear function can gain more generalization ability at the cost of more weights and computation. The achieves the best balance between the performance and cost when the number of heads is 2 for DeepEnFM. The of AutoInt show that the influence of head number is smaller than DeepEnFM, because it uses scale dot product function, which is less sensitive to the dimension of query and key vectors. In this paper, we propose a novel framework named Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM), which aims to learn a better aligned vector embedding through the encoder. The encoder combines the bilinear attention and max-pooling method to gather both the complementary and suppressing information from the content of other fields. The extensive experiments demonstrate that our approach achieves state-of-art performance on Criteo and Avazu dataset.

DNN and Encoder enhanced FM with bilinear attention and max-pooling for CTR

Represent the TLDR for finding the corresponding scientific article.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.