Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We present an approach for anytime predictions in deep neural networks (DNNs). For each test sample, an anytime predictor produces a coarse quickly, and then continues to refine it until the test-time computational budget is depleted. Such predictors can address the growing computational problem of DNNs by automatically adjusting to varying test-time budgets. In this work, we study a \emph{general} augmentation to feed-forward networks to form anytime neural networks (ANNs) via auxiliary predictions and losses. Specifically, we point out a blind-spot in recent studies in such ANNs: the importance of high final accuracy. In fact, we show on multiple recognition data-sets and architectures that by having near-optimal final predictions in small anytime models, we can effectively double the speed of large ones to reach corresponding accuracy level. We achieve such speed-up with simple weighting of anytime losses that oscillate during training. We also assemble a sequence of exponentially deepening ANNs, to achieve both theoretically and practically near-optimal anytime at any budget, at the cost of a constant fraction of additional consumed budget. In recent years, the accuracy in visual recognition tasks has been greatly improved by increasingly complex convolutional neural networks, from AlexNet BID8 and VGG BID12, to ResNet BID3, ResNeXt BID14, and DenseNet BID6. However, the number of applications that require latency sensitive responses is growing rapidly. Furthermore, their test-time computational budget can often. E.g., autonomous vehicles require real-time object detection, but the required detection speed depends on the vehicle speed; web servers need to meet varying amount of data and user requests throughput through out a day. Thus, it can be difficult for such applications to choose between slow predictors with high accuracy and fast predictors with low accuracy. In many cases, this dilemma can be resolved by an anytime predictor BID4 BID0 BID16, which, for each test sample, produces a fast and crude initial prediction and continues to refine it as budget allows, so that at any test-time budget, the anytime predictor has a valid for the sample, and the more budget is spent, the better the prediction is. In this work 1, we focus on the anytime prediction problem in neural networks. We follow the recent works BID10 BID15 BID5 to append auxiliary predictions and losses in feed-forward networks for anytime predictions, and train them jointly end-to-end. However, we note that the existing methods all put only a small fraction of the total weightings to the final prediction, and as a , large anytime models are often only as accurate as much smaller non-anytime models, because the accuracy gain is so costly in DNNs, as demonstrated in FIG0. We address this problem with a novel and simple oscillating weightings of the losses, and will show in Sec. 3 that our small anytime models with near-optimal final predictions can effectively speed up two times large ones without them, on multiple data-sets, including ILSVRC BID11, and on multiple models, including the very recent Multi-ScaleDenseNets (MSDnets) BID5. Observing that the proposed training techniques lead to ANNs that are near-optimal in late predictions but are not as accurate in the early predictions, we assemble ANNs of exponentially increasing depths to dedicate early predictions to smaller networks, while only delaying large networks by a constant fraction of additional test-time budgets. As illustrated in FIG0, given a sample (x, y) ∼ D, the initial feature map x 0 is set to x, and the subsequent feature transformations f 1, f 2,..., f L generate a sequence of intermediate features DISPLAYFORM0 Each feature map x i can then produce an auxiliary predictionŷ i using a prediction layer g i:ŷ i = g i (x i ; w i) with parameter w i. Each auxiliary predictionŷ i then incurs an expected DISPLAYFORM1. We call such an augmented network as an Anytime Neural Network (ANN). Let the parameters of the full ANN be θ = (DISPLAYFORM2 The most common way to optimize these losses, 1, ..., L, end-to-end is to optimize them in a weighted sum min θ L i=1 B i i (θ), where {B i} i form the weight scheme for the losses. Alternating SIEVE weights. Three experimental observations lead to our proposed SIEVE weight scheme. First, the existing weights, CONST BID10 BID5, and LINEAR both incur more than 10% relative increase in final test errors, which effectively slow down anytime models multiple times. Second, we found that a large weight can improve a neighborhood of losses thanks to the high correlation among neighboring losses. Finally, keeping a fixed weighting may lead to solutions where the sum of the gradients are zero, but the individual gradients are non-zero. The proposed SIEVE scheme has half of the total weights in the final loss, so that the final gradient can outweigh other gradients when all loss gradients have equal two-norms. It also have uneven weights in early losses to let as many losses to be near large weights as possible. Formally for L losses, we first add to B L 2 one unit of weight, where • means rounding. We then add one unit to each B kL 4 for k = 1, 2, 3, and then to each B kL 8 for k = 1, 2,..., 7, and so on, until all predictors have non-zero weights. We finally normalize B i so that DISPLAYFORM3, and set B L = 1. During each training iteration, we also sample proportional to the B i a layer i, and add temporarily to the total loss B L i so as to oscillate the weights to avoid spurious solutions. We call ANNs with alternating weights as alternating ANNs (AANNs). Though the proposed techinques are heuristics, they effectively speed up anytime models multiple times as shown in Sec. 3. We hope our experimental can inspire, and set baselines for, future principled approaches. EANN. Since AANNs put high weights in the final layer, they trade early accuracy for the late ones. We leverage this effect to improving early predictions of large ANNs: we propose to form a sequence of ANNs whose depths grow exponentially (EANN). By dedicating early predictions to small networks, EANN can achieve better early . Furthermore, if the largest model has L depths, we only compute log L small networks before the final one, and the total cost of the small networks is only a constant fraction of the final one. Hence, we only consume a constant fraction of additional test-time budget. FIG0 how an EANN of the exponential base b = 2 works at test-time. The EANN sequentially computes the ANNs, and only outputs an anytime if the current is better than previous ones in validation. Formally, if we assume that each ANN has near-optimal after 1 b of its layers, then we can prove that for any budget B, the EANN can We present two key : small anytime models with SIEVE can outperform large ones with CONST, and EANNs can improve early accuracy, but cost a constant fraction of extra budgets. SIEVE vs. CONST of double costs. In FIG1 and FIG1, we compare SIEVE and CONST on ANNs that are based on ResNets on CIFAR100 BID7 ) and ILSVRC BID11. The networks with CONST have double the depths as those with SIEVE. We observe that SIEVE leads to the same final error rates as CONST of double the costs, but does so much faster. The two schemes also have similar early performance. Hence, SIEVE effectively speed up the predictions of CONST by about two times. In FIG1, we experiment with the very recent Multi-Scale-DenseNets (MSDNets) BID5, which are specifically modified from the recently popular DenseNets BID6 to produce the state-of-the-art anytime predictions. We again observe that by improving the final anytime prediction of the smallest MSDNet26 without sacrificing too much early predictions, we make MSDNet26 effectively a sped-up version of MSDNet36 and MSDNet41.EANN vs. ANNs and OPT. In FIG1, we assemble ResNet-ANNs of 45, 81 and 153 conv layers to form EANNs. We compare the EANNs against the parallel OPT, which is from running regular networks of various depths in parallel. We observe that EANNs are able to significantly reduce the early errors of ANNs, but reach the final error rate later. Furthermore, ANNs with more accurate final predictions using SIEVE and EXP-LIN 2 are able to outperform CONST and LINEAR, since whenever an ANN completes in an EANN, the final is the best one for a long period of time.

By focusing more on the final predictions in anytime predictors (such as the very recent Multi-Scale-DenseNets), we make small anytime models to outperform large ones that don't have such focus.


Question: Represent the scientific article to find a short TLDR

Question: Computational imaging systems jointly design computation and hardware to retrieve information which is not traditionally accessible with standard imaging systems. Recently, critical aspects such as experimental design and image priors are optimized through deep neural networks formed by the unrolled iterations of classical physics-based reconstructions (termed physics-based networks). However, for real-world large-scale systems, computing gradients via backpropagation restricts learning due to memory limitations of graphical processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network’s layers to enable data-driven design for large-scale computational imaging. We demonstrate our methods practicality on two large-scale systems: super-resolution optical microscopy and multi-channel magnetic resonance imaging. Computational imaging systems (tomographic systems, computational optics, magnetic resonance imaging, to name a few) jointly design software and hardware to retrieve information which is not traditionally accessible on standard imaging systems. Generally, such systems are characterized by how the information is encoded (forward process) and decoded (inverse problem) from the measurements. The decoding process is typically iterative in nature, alternating between enforcing data consistency and image prior knowledge. Recent work has demonstrated the ability to optimize computational imaging systems by unrolling the iterative decoding process to form a differentiable Physicsbased Network (PbN) (1; 2; 3) and then relying on a dataset and training to learn the system's design parameters, e.g. experimental design (3; 4; 5), image prior model (1; 2; 6; 7). PbNs are constructed from the operations of reconstruction, e.g. proximal gradient descent algorithm. By including known structures and quantities, such as the forward model, gradient, and proximal updates, PbNs can be efficiently parameterized by only a few learnable variables, thereby enabling an efficient use of training data while still retaining robustness associated with conventional physics-based inverse problems. Training PbNs relies on gradient-based updates computed using backpropagation (an implementation of reverse-mode differentiation ). Most modern imaging systems seek to decode ever-larger growing quantities of information (gigabytes to terabytes) and as this grows, memory required to perform backpropagation is limited by the memory capacity of modern graphical processing units (GPUs). Methods to save memory during backpropagation (e.g. forward recalculation, reverse recalculation, and checkpointing) trade off spatial and temporal complexity. For a PbN with N layers, standard backpropagation achieves O(N) temporal and spatial complexity. Forward recalculation achieves O memory complexity, but has to recalculate unstored variables forward from the input of the network when needed, yielding O(N 2) temporal complexity. Forward checkpointing smoothly trades off temporal, O(N K), and spatial, O(N/K), complexity by saving variables every K layers and forward-recalculating unstored variables from the closest checkpoint. Reverse recalculation provides a practical solution to beat the trade off between spatial vs. temporal complexity by calculating unstored variables in reverse from the output of the network, yielding O(N) temporal and O spatial complexities. Recently, several reversibility schemes have been proposed for residual networks, learning ordinary differential equations, and other specialized network architectures (11; 12). In this work, we propose a memory-efficient learning procedure for backpropagation for the PbN formed from proximal gradient descent, thereby enabling learning for many large-scale computational imaging systems. Based on the concept of invertibility and reverse recalculation, we detail how backpropagation can be performed without the need to store intermediate variables for networks composed of gradient and proximal layers. We highlight practical restrictions on the layers and introduce a hybrid scheme that combines our reverse recalculation methods with checkpointing to mitigate numerical error accumulation. Finally, we demonstrate our method's usefulness to learn the design for two practical large-scale computational imaging systems: superresolution optical microscopy (Fourier Ptychography) and multi-channel magnetic resonance imaging. Computational imaging systems are described by how sought information is encoded to and decoded from a set of measurements. The encoding of information, x into measurements, y, is given by where A is the forward model that characterizes the measurement system physics and n is random system noise. The forward model is a continuous process, but is often approximated by a discrete representation. The retrieval of information from a set of measurements, i.e. decoding, is commonly structured using an inverse problem formulation, where D(·) is a data fidelity penalty and P(·) is a prior penalty. When n is governed by a known noise model, the data consistency penalty can be written as the negative log-likelihood of the appropriate distribution. When P(·) is a non-smooth prior (e.g. 1, total variation), proximal gradient descent (PGD) and its accelerated variants are often efficient algorithms to minimize the objective in Eq. 2 and are composed of the following alternating steps: where α is the gradient step size, ∇ x is the gradient operator, prox P is a proximal function that enforces the prior, and x (k) and z (k) are intermediate variables for the k th iteration. The structure of the PbN is determined by unrolling N iterations of the optimizer to form the N layers of a network (Eq. 3 and Eq. 4 form a single layer). Specifically, the input to the network is the initialization of the optimization, x, and the output is the ant, x (N). The learnable parameters are optimized using gradient-based methods. Common machine learning toolboxes' (e.g. PyTorch, Tensor Flow, Caffe) auto-differentiation functionalities are used to compute gradients for backpropagation. Auto-differentiation accomplishes this by creating a graph composed of the PbN's operations and storing intermediate variables in memory. Our main contribution is to improve the spatial complexity of backpropagation for PbNs by treating the larger single graph for auto-differentiation as a series of smaller graphs. Specifically, consider a PbN, F, composed of a sequence of layers, where x (k) and x (k+1) are the k th layer input and output, respectively, and θ (k) are its learnable parameters. When performing reverse-mode differentiation, our method treats a PbN of N layers as N separate smaller graphs, processed one at a time, rather than as a single large graph, thereby saving a factor N in memory. As outlined in Alg. 1, we first recalculate the current layer's input, inverse, and then form one of the smaller graphs by recomputing the output of the layer, v (k), from the recalculated input. To compute gradients, we then rely on auto-differentiation of each layer's smaller graph to compute the gradient of the loss, L, with respect to The procedure is repeated for all N layers in reverse order. Algorithm 1 Memory-efficient learning for physics-based networks 1: procedure MEMORY-EFFICIENT BACKPROPAGA- for k > 0 do 4: end for 10: 11: end procedure In order to perform the reverse-mode differentiation efficiently, we must be able to compute each layer's inverse operation, inverse. The remainder of this section overviews the procedures to invert gradient and proximal update layers. A common interpretation of gradient descent is as a forward Euler discretization of a continuous-time ordinary differential equation. As a consequence, the inverse of the gradient step layer (Eq. 3) can be viewed as a backward Euler step, This implicit equation can be solved iteratively via the backward Euler method using the fixed point algorithm (Alg. 2). Convergence is guaranteed if where Lip(·) computes the Lipschitz constant of its argument. In the setting when D(x; y) = Ax − y 2 and A is linear this can be ensured if α <, where σ max (·) computes the largest singular value of its argument. Finally, as given by Banach Fixed Point Theorem, the fixed point algorithm (Alg. 2) will have an exponential rate of convergence. Algorithm 2 Inverse for gradient layer 1: procedure FIXED POINT METHOD(z, T) 2: x ← z 3: for t < T do 4: x ← z + α∇ x D(x; y) 5: end for return x 8: end procedure The proximal update (Eq. 4) is defined by the following optimization problem: For differentiable P(·), the optimum of which is, In contrast to the gradient update layer, the proximal update layer can be thought of as a backward Euler step. This allows its inverse to be expressed as a forward Euler step, when the proximal function is bijective (e.g. prox). If the proximal function is not bijective (e.g. prox 1) the inversion is not straight forward. However, in many cases it is possible to substitute it with a bijective function with similar behavior. Reverse recalculation of the unstored variables is non-exact as the operations to calculate the variables are not identical to forward calculation. The is numerical error between the original forward and reverse calculated variables and as more iterations are unrolled, numerical error can accumulate. To mitigate these effects, some of the intermediate variables can be stored from forward calculation, referred to as checkpoints. Memory permitting, as many checkpoints should be stored as possible to ensure accuracy while performing reverse recalculation. While most PbNs cannot afford to store all variables required for reverse-mode differentiation, it is often possible to store a few. Standard bright-field microscopy offers a versatile system to image in vitro biological samples, however, is restricted to imaging either a large field of view or a high resolution. Fourier Ptychographic Microscopy (FPM) is a super resolution (SR) method that can create gigapixel-scale images beating this trade off on a standard optical microscope by acquiring a series of measurements (up to hundreds) under various illumination settings on an LED array microscopy and combining them via a phase retrieval based optimization. The system's dependence on many measurements inhibits its ability to image live fast-moving biology. Reducing the number of measurements is possible using linear multiplexing and state of the art performance is achieved by forming a PbN and learning its experimental design (4; 3), however, is currently limited in scale due to GPU memory constraints (terabyte-scale memory is required for learning the full measurement system). With our proposed memory-efficient learning framework, we reduce the required memory to only a few gigabytes, thereby enabling the use of consumer-grade GPU hardware. To evaluate accuracy we compare standard learning with our proposed memory-efficient learning on a problem that fits in standard GPU memory. We reproduce in where the number of measurements are reduced by a factor of 10 using 6.26GB of memory using only 0.627GB and time is only increased by a factor of 2. To perform memory-efficient learning, we set T = 4 and checkpoint every 10 unrolled iterations. The testing loss between our method and standard learning are comparable (Fig. 1a). In addition, we qualitatively highlight equivalence of the two methods, displaying SR reconstructions with learned design using standard (Fig. 1d) and memory-efficient (Fig. 1e) methods. For relative comparison, we display a single low resolution measurement (Fig. 1b) and the ground truth SR reconstruction using all measurements (Fig. 1c). MRI is a powerful Fourier-based medical imaging modality that non-invasively captures rich biophysical information without ionizing radiation. Since MRI acquisition time is directly proportional to the number of acquired measurements, reducing measurements leads to immediate impact on patient throughput and enables capturing fast-changing physiological dynamics. Multi-channel MRI is the standard of care in clinical systems and uses multiple receive coils distributed around the body to acquire measurements in parallel, thereby reducing the total number of required acquisition frames for decoding. By additionally modifying the measurement pattern to take advantage of image prior knowledge, e.g. through compressed sensing, it is possible to dramatically reduce scan times. As with experimental design, PbNs with learned deep image priors have demonstrated state-of-the-art performance for multi-channel MRI (20; 6), but are limited in network size and number of unrolled iterations due to memory required for training. Our memory-efficient learning reduces memory footprint at training time, thereby enabling learning for larger problems. To evaluate our proposed memory-efficient learning, we reproduce the in for the "SD-ET-WD" PbN, which is equivalent to PGD (10 unrolled iterations) where the proximal update is replaced with a learned invertible residual convolutional neural network (RCNN) (21; 11; 9). We compare training with full backpropagation, requiring 10.77GB of memory and 3:50 hours, versus memory-efficient learning, requiring 2.11GB and 8:25 hours. We set T = 6 and do not use checkpointing. As Fig. 2 shows, the training loss is comparable across epochs, and inference are similar on one image in the training set, with normalized root mean-squared error of 0.03 between conventional and memory-efficient learning. Discussion: Our proposed memory-efficient learning opens the door to applications that are not otherwise possible to train due to GPU memory constraints, without a large increase in training time. While we specialized the procedure to PGD networks, similar approaches can be taken to invert other PbNs with more complex subroutines such as solving linear systems of equations. However, sufficient conditions for invertibility must be met. This limitation is clear in the case of a gradient descent block with an evolving step size, as the Lipschitz constant may no longer satisfy Eq. 7. Furthermore, the convergent behavior of optimization to minima makes accurate reverse recalculation of unstored variables severely ill-posed and can cause numerical error accumulation. Checkpoints can be used to improve the accuracy of reverse recalculated variables, though most PbN are not deep enough for numerical convergence to occur. In this communication, we presented a practical memory-efficient learning method for large-scale computational imaging problems without dramatically increasing training time. Using the concept of reversibility, we implemented reverse-mode differentiation with favorable spatial and temporal complexities. We demonstrated our method on two representative applications: SR optical microscopy and multi-channel MRI. We expect other computational imaging systems to nicely fall within our framework.

represent the tldr for finding the corresponding scientific article!

We propose a memory-efficient learning procedure that exploits the reversibility of the network’s layers to enable data-driven design for large-scale computational imaging.

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the article for finding a single-sentence TLDR

Question: Existing multi-agent reinforcement learning (MARL) communication methods have relied on a trusted third party (TTP) to distribute reward to agents, leaving them inapplicable in peer-to-peer environments. This paper proposes reward distribution using {\em Neuron as an Agent} (NaaA) in MARL without a TTP with two key ideas: (i) inter-agent reward distribution and (ii) auction theory. Auction theory is introduced because inter-agent reward distribution is insufficient for optimization. Agents in NaaA maximize their profits (the difference between reward and cost) and, as a theoretical , the auction mechanism is shown to have agents autonomously evaluate counterfactual returns as the values of other agents. NaaA enables representation trades in peer-to-peer environments, ultimately regarding unit in neural networks as agents. Finally, numerical experiments (a single-agent environment from OpenAI Gym and a multi-agent environment from ViZDoom) confirm that NaaA framework optimization leads to better performance in reinforcement learning. To the best of our knowledge, no existing literature discusses reward distributions in the configuration described above. Because CommNet assumes an environment that distributes a uniform reward to all the agents, if the distributed reward is in limited supply (such as money), it causes the Tragedy of the Commons BID15, where the reward of contributing agents will be reduced due to the participation of free riders. Although there are several MARL methods for distributing rewards ac- BID0 BID23 BID6 BID7. They should suppose TTP to distribute the optimal reward to the agents. (b) Inter-agent reward distribution model (our model). Some agents receive reward from the environment directly, and redistribute to other agents. The idea to determine the optimal reward without TTP is playing auction game among the agents.cording to agents' contribution such as QUICR BID0 and COMA BID23, they suppose the existence of TTP and hence cannot be applied to the situation investigated here. The proposed method, Neuron as an Agent (NaaA), extends CommNet to actualize reward distributions in MARL without TTP based on two key ideas: (i) inter-agent reward distribution and (ii) auction theory. Auction theory was introduced because inter-agent reward distributions were insufficient for optimization. Agents in NaaA maximize profit, the difference between their received rewards and the costs which they redistribute to other agents. If the framework is naively optimized, a trivial solution is obtained where agents reduce their costs to zero to maximize profits. Then, NaaA employs the auction theory in game design to prevent costs from dropping below their necessary level. As a theoretical , we show that agents autonomously evaluate the counterfactual return as values of other agents. The counterfactual return is equal to the discounted cumulative sum of counterfactual reward BID0 distributed by QUICR and COMA.NaaA enables representation trades in peer-to-peer environments and, ultimately, regards neural network units as agents. As NaaA is capable of regarding units as agents without losing generality, this setting was utilized in the current study. The concept of the proposed method is illustrated in FIG0.An environment extending ViZDoom BID11, a POMDP environment, to MARL was used for the experiment. Two agents, a cameraman sending information and a main player defeating enemies with a gun, were placed in the environment. Results confirmed that the cameraman learned cooperative actions for sending information from dead angles (behind the main player) and outperformed CommNet in score. Interestingly, NaaA can apply to single-and multi-agent settings, since it learns optimal topology between the units. Adaptive DropConnect (ADC), which combines DropConnect (randomly masking topology) with an adaptive algorithm (which has a higher probability of pruning connections with lower counterfactual returns) was proposed as a further application for NaaA. Experimental classification and reinforcement learning task showed ADC outperformed DropConnect. The remainder of this paper is organized as follows. In the next section, we show the problem setting. Then, we show proposed method with two key ideas: inter-agent reward distribution and auction theory in Section 3. After related works are introduced in Section 4, the experimental are shown in classification, single-agent RL and MARL in Section 5. Finally, a ends the paper. Suppose there is an N -agent system in an environment. The goal of this paper was to maximize the discounted cumulative reward the system obtained from the environment. This was calculated as: DISPLAYFORM0 where R ex t is a reward which the system obtains from the environment at t, and γ ∈ is the discount rate and T is the terminal time. Reward distribution is distributing R t to all the agents under the following constraint. DISPLAYFORM1 where R it is a reward which is distributed to i-th agent at time t. For instance, in robot soccer, the environment give a reward 1 when an agent shoot a ball to the goal. Each agent should receive the reward along to their contribution. In most of MARL communication methods, the policy of reward distribution is determined by a centralized agent. For example, QUICR BID0 and COMA BID7 distribute R it according to counterfactal reward, difference of reward between an agent made an action and not. The value of counterfactual reward is calculated by centralized agent, called trusted third party (TTP).In a peer-to-peer environment such as inter-industry and -country trade, they cannot place a TTP. Hence, another framework required to actualize reward distribution without TTP. The proposed method, Neuron as an Agent (NaaA), extends CommNet BID23 to actualize reward distributions in MARL without TTP based on two key ideas: (i) inter-agent reward distribution and (ii) auction theory. As we show in Section 3.3, NaaA actualizes that we can regard even a unit as an agent. Some agents receive rewards from the environment directly R ex it, and they distribute these to other agents as incentives for giving precise information. Rewards are limited, so if an agent distributes ρ rewards, the reward of that agents is reduced by ρ to satisfy the constraint of Eq:. For this reason, agents other than a specified agent of interest can be considered a secondary environment for the agent giving rewards of −ρ instead of an observation x. This secondary environment was termed the internal environment, whereas the original environment was called the external environment. Similarly to CommNet BID23, the communication protocol between agents was assumed to be a continuous quantity (such as a vector), the content of which could be trained by backpropagation. A communication network among the agents is represented as a directed graph G = (V, E) between agents, where V = {v 1, . . ., v N} is a set of the agents and E ⊂ V 2 is a set of edges representing the connections between two agents. If (v i, v j) ∈ E, then connection v i → v j holds, indicating that v j observes the representation of v i. Here, the representation of agent v i at time t was denoted as x it ∈ R. Additionally, the set of agents that agent i connects to was designated to be N out i = {j|(v i, v j) ∈ E} and the set of agents that agent i is connected from was DISPLAYFORM0 The following assumptions were added to the v i characteristics: N1: (Selfishness) The utility each agent v i wants to maximize is its own return (cumulative discounted reward): DISPLAYFORM1 N2: (Conservation) The summation of internal rewards over all V equals 0. Hence, the summation of rewards which V (receive both internal and external environment R it) are equivalent to the reward R ex t, which the entire multi-agent system receives from the external environment: DISPLAYFORM2 representation signal x i before transferring this signal to the agent. Simultaneously, ρ jit will be subtracted from the reward of v j. N4: (NOOP) v i can select NOOP (no operation), for which the return is δ > 0, as an action. In NOOP, the agent inputs and outputs nothing. The social welfare function (total utility of the agents) G all is equivalent to the objective function G. That is, DISPLAYFORM3 From N2, G all = G holds. From N3, the reward R it received by v i at t can be written as: DISPLAYFORM0 which can be divided into positive and negative terms, where the former is defined as revenue, and the latter as cost. These are respectively denoted as DISPLAYFORM1 Here, R it represents profit, difference between revenue and cost. The agent v i maximizes its cumulative discounted profit, G it, represented as: DISPLAYFORM2 G it could not be observed until the end of an episode (the final time). Because predictions based on current values were needed to select optimal actions, G it was approximated with the value function DISPLAYFORM3 where s it is a observation for i-th agent at time t. Under these conditions, the following equation holds: DISPLAYFORM4 Thus, herein, we only consider maximization of revenue, the value function, and cost minimization. The inequality R it > 0 (i.e., r it > c it) indicates that the agent in question gave additional value to the obtained data. The agent selected the NOOP action because DISPLAYFORM5 If we directly optimize Eq:, a trivial solution is obtained in which the internal rewards converge at 0, and all agents (excepting agents which directly receive reward from the external environment) select NOOP as their action. This phenomenon occurs regardless of the network topology G, as no nodes are incentivized to send payments ρ ijt to other agents. With this in mind, multi-agent systems must select actions with no information, achieving the equivalent of taking random actions. For that reason, the total external reward R ex t shrinks markedly. This phenomenon also known as social dilemma in MARL, which is caused from a problem that each agent does not evaluate other agents' value truthfully to maximize their own profit. We are trying to solve this problem with auction theory in Section 3.2. To make the agents to evaluate other agents' value truthfully, the proposed objective function borrows its idea from the digital goods auction theory BID10. In general, an auction theory is a part of mechanism design intended to unveil the true price of goods. Digital goods auctions are one mechanism developed from auction theory, specifically targeting goods that may be copied without cost such as digital books and music. Although several variations of digital goods auctions exist, an envy-free auction BID10 was used here because it required only a simple assumption: equivalent goods have a single simultaneous price. In NaaA, this can be represented by the following assumption: DISPLAYFORM0 The assumption above indicates that ρ jit takes either 0 or a positive value, depending on i at an equal selected time t. Therefore, the positive side was named v i's price, and denoted as q it.The envy-free auction process is shown in the left section of FIG2, displaying the negotiation process between one agent sending an representation (defined as a seller), and a group of agents buying the representation (defined as buyers). First, a buyer places a bid with the agent at a bidding price b jit. Next, the seller selects the optimal priceq it and allocates the representation. Payment occurs if b ijt exceeds q jt. In this case, ρ jit = H(b jit − q it)q it holds where H(·) is a step function. For this transaction, the definition g jit = H(b jit − q it) holds, and is named allocation. After allocation, buyers perform payment: ρ jit = g jitqit. The seller sends the representation x i only to the allocated buyers. Buyers who do not receive the representation approximate DISPLAYFORM1 This negotiation is performed at each time step in reinforcement learning. The sections below discuss the revenue, cost, and value functions based on Eq:.Revenue: The revenue of an agent is given as DISPLAYFORM0 where DISPLAYFORM1 g jit is the demand, the number of agents for which the bidding price b jit is greater than or equal to q it. Because R ex i is independent of q it, the optimal priceq it maximizing r it is given as:q DISPLAYFORM2 The r it curve is shown on the right side of FIG2.Cost: Cost is defined as an internal reward that one agent pays to other agents. It is represented as: DISPLAYFORM3 where The effects of the value function were considered for both successful and unsuccessful v j purchasing cases. The value function was approximated as a linear function g it: DISPLAYFORM4 DISPLAYFORM5 where o it is equivalent to the cumulative discount value of the counterfactual reward BID0, it was named counterfactual return. As V 0 it (a constant independent of g it) is equal to the value function when v i takes an action without observing x 1,..., x N.The optimization problem is, therefore, presented below using a state-action value function for i-th where DISPLAYFORM6 DISPLAYFORM7 was taken because the asking priceq t was unknown for v i, except whenq it and g iit = 0.Then, to identify the bidding price that b it maximizes returns, the following theorem holds. This proof is shown in the Appendix A.This implies agents should only consider their counterfactual returns! When γ = 0 it is equivalent to a case without auction. Hence, the bidding value is raised if each agent considers their long-time rewards. Consequently, when the NaaA mechanism is used agents behave as if performing valuation for other agents, and declare values truthfully. Under these conditions, the following corollary holds:Corollary 3.1. The Nash equilibrium of an envy-free auction is DISPLAYFORM0 The remaining problem is how to predict o t. Q-learning was used to predict o t in this paper as the same way as QUICR BID0. As o it represented the difference between two Qs, each Q was approximated. The state was parameterized using the vector s t, which contained input and weight. The ϵ-greedy policy with Q-learning typically supposed that discrete actions Thus the allocation g ijt was employed as an action rather than b it and q it.Algorithm The overall algorithm is shown in Algorithm 1. One benefit of NaaA is that it can be used not only for MARL, but also for network training. Typical neural network training algorithms such as RMSProp and Adam BID13 are based on sequential algorithms such as the stochastic gradient descent (SGD). Therefore, the problem they solve can be interpreted as a problem of updating a state (i.e., weight) to a goal (the minimization of the expected likelihood). Learning can be accelerated by applying NaaA to the optimizer. In this paper, the application of NaaA to SGD was named Adaptive DropConnect (ADC), the finalization of which can be interpreted as a combination of DropConnect and Adaptive DropOut BID2. In the subsequent section, ADC is introduced as a potential NaaA application. Algorithm 1 NaaA: inter-agent reward distribution with envy-free auction 1: for t = 1 to T do 2:Compute a bidding price for every edge: DISPLAYFORM0 Compute an asking price for every node: DISPLAYFORM1 qd it (q). for DISPLAYFORM0 Compute allocation: DISPLAYFORM1 Compute the price the agent should pay: ρ jit ← g jitqit 7:end for 8:Make a payment: DISPLAYFORM2 Make a shipment: DISPLAYFORM3 for v i ∈ V do Compute a bidding price for every edge: DISPLAYFORM4 Compute an asking price for every node: DISPLAYFORM5 qd it (q). for DISPLAYFORM0 Compute allocation: DISPLAYFORM1 end for Sample a switching matrix U t from a Bernoulli distribution: DISPLAYFORM0 Sample the random mask M t from a Bernoulli distribution: DISPLAYFORM1 Generate the adaptive mask: DISPLAYFORM2 Compute h t for making a shipment: DISPLAYFORM3 Update W t and b t by backpropagation. 12: end for ADC uses NaaA for supervised optimization problems with multiple revisions. In such problems, the first step is the presentation of an input state (such as an image) by the environment. Agents are expected to update their parameters to maximize the rewards presented by a criterion calculator. Criterion calculators gives batch-likelihoods to agents, representing rewards. Each agent, a classifier, updates its weights to maximize the reward from the criterion calculator. These weights are recorded as an internal state. A heuristic utilizing the absolute value of weight |w ijt | (the technique used by Adaptive DropOut) was applied as the counterfactual return o ijt. The absolute value of weights was used because it represented the updated amounts for which the magnitude of error of unit outputs was proportional to |w ijt |.This algorithm is presented as Algorithm 2. Because the algorithm is quite simple, it can be easily implemented and, thus, applied to most general deep learning problems such as image recognition, sound recognition, and even deep reinforcement learning. Existing multi-agent reinforcement learning (MARL) communication methods have relied on a trusted third party (TTP) to distribute reward to agents, leaving them inapplicable in peer-to-peer environments. R/DIAL BID6 ) is a communication method for deep reinforcement learning, which train the optimal communication among the agent with Q-learning. It focuses on that paradigm of centralized planning. CommNet BID23, which exploits the characteristics of a unit that is agnostic to the topology of other units, employs backpropagation to train multi-agent communication. Instead of reward R(a t) of an agent i for actions at t a t, QUICR-learning BID0 ) maximizes counterfactual reward R(a t) − R(a t − a it), the difference in the case of the agent i takes an action a it (a t) and not (a t − a it). COMA BID7 ) also maximizes counterfactual rewards in an actor-critic setting. CommNet, QUICR and COMA have a centralized environment for distributing rewards through a TTP. In contrast, NaaA does not rely on a TTP, and hence, each agent calculates its reward. While inter-agent reward distribution has not been considered in the context of communication, trading agents have been considered in other contexts. Trading agent competition (TACs), competitions for trading agent design, have been held in various locations regarding topics such as smart grids BID12, wholesale BID14, and supply chains BID19, yielding innumerable trading algorithms such as Tesauro's bidding algorithm BID26 and TacTex'13 . Since several competitions employed an auction as optimal price determination mechanism (; BID22, using auctions to determine optimal prices is now a natural approach. Unfortunately, these existing methods cannot be applied to the present situation. First, their agents did not communicate because the typical purpose of a TAC is to create market competition between agents in a zero-sum game. Secondly, the traded goods are not digital goods but instead goods in limited supply, such as power and tariffs. Hence, this is the first paper to introduce inter-agent reward distribution to MARL communications. Auction theory is discussed in terms of mechanism design BID17, also known as inverse game theory. Second-price auctions are auctions including a single product and several buyers. In this paper, a digital goods auction BID10 was used as an auction with an infinite supply. Several methods extend digital goods auction to address collusion, including the consensus estimate BID8 and random sample auction BID9, which can be used to improve our method. This paper is also related to DropConnect in terms of controlling connections between units. Adaptive DropConnect (ADC), proposed in a later section of this paper as a further application, extends the DropConnect regularization technique. The finalized idea of ADC (which uses a skew probability correlated to the absolute value of weights rather than dropping each connection between units by a constant probability) is closer to Adaptive DropOut BID2, although their derivation differs. The adjective "adaptive" is added with respect to the method. Neural network optimizing using RL was investigated by BID1; however, their methods used a recurrent neural network (RNN) and are therefore difficult to implement, whereas the proposed method is RNN-free and forms as a layer. For these reasons, its implementation is simple and fast and it also has a wide area of applicability. To confirm that NaaA works widely with machine learning tasks, we confirm our method of supervised learning tasks as well as reinforcement learning tasks. As supervised learning tasks, we use typical machine learning tasks such as image classification using MNIST, CIFAR-10, and SVHN.As reinforcement tasks, we confirm single-and multi-agent environment. The single-agent environment is from OpenAI Gym. We confirm the using a simple reinforcement task: CartPole. In multi-agent, we use ViZDoom, a 3D environment for reinforcement learning. For classification, three types of datasets were used: MNIST, CIFAR-10, and STL-10. The given task was to predict the label of each image, and each dataset had a class number of 10. The first dataset, MNIST, was a collection of black and white images of handwritten digits sized 28 28. The training and test sets contained 60,000 and 10,000 example images, respectively. The CIFAR-10 dataset images were colored and sized 32 32, and the assigned task was to predict what was shown in each picture. This dataset contained 6,000 images per class (5,000 for training and 1,000 for testing). The STL-10 dataset was used for image recognition, and had 1,300 images for each class (500 training, 800 testing). Each image was sized 96 96; however, for the experiment, the images were resized to 48 48 because the greater resolution of this dataset (relative to the above datasets) required far more computing time and resources. Two models were compared in this experiment: DropConnect and Adaptive DropConnect (the model proposed in this paper). The baseline model was composed of two convolutional layers and two fully connected layers whose outputs are dropped out (we set the possibility as 0.5). The labels of input data were predicted using log-softmaxed values from the last fully connected layer. In the DropConnect and Adaptive DropConnect models, the first fully connected layer was replaced by a DropConnected and Adaptive DropConnected layer, respectively. It should be noted that the DropConnect model corresponded to the proposed method when ε = 1.0, meaning agents did not perform their auctions but instead randomly masked their weights. The models were trained over ten epochs using the MNIST datasets, and were then evaluated using the test data. The CIFAR-10 and STL-10 epoch numbers were 20 and 40, respectively. Experiments were repeated 20 times for each condition, and the averages and standard deviations of error rates were calculated. Results are shown in TAB1. As expected, the Adaptive DropConnect model performed with a lower classification error rate than either the baseline or DropConnect models regardless of the given experimental datasets. Next, the single-agent reinforcement learning task was set as the CartPole task from OpenAI Gym BID3 with visual inputs. In this setting, the agent was required to balance a pole while moving a cart. The images contained a large amount of non-useful information, making pixel pruning important. The in TAB1 demonstrates that our method improves the standard RL. The proposed reward distribution method was confirmed to work as expected by a validation experiment using the multi-agent setting in ViZDoom BID11, an emulator of Doom containing a map editor where additional agents complement the main player. A main player in the ViZDoom environment aims to seek the enemy in the map and then defeat the enemy. A defend the center (DtC)-based scenario, provided by ViZDoom platform, was used for this experiment. Two players, a main player and a cameraman, were placed in the DtC, where they started in the center of a circular field and then attacked enemies that came from the surrounding wall. Although the main player could attack the enemy with bullets, the cameraman had no way to attack, only scouting for the enemy. The action space for the main player was the combination of {attack, turn left, turn right}, giving a total number of actions 2 3 = 8. The cameraman had two possible actions: {turn left, turn right}. Although the players could change direction, they could not move on the field. Enemies died after receiving one attack (bullet) from the main player, and then player received a score of +1 for each successful attack. The main player received 26 bullets by default at the beginning of each episode. The main player died if they received attacks from the enemy to the extent that their health dropped to 0, and received a score of -1 for each death. The cameraman did not die if attacked by an enemy. Episodes terminated either when the maim player died or after 525 steps elapsed. Figure 4: NaaA leads agents to enter a cooperative relationship. First, the two agents face different directions, and the cameraman sells their information to the main player. The main player (information buyer) starts to turn right to find the enemy. The cameraman (information seller) starts to turn left to seek new information by finding the blind area of the main player (2 and 3). After turning, the main player attacks the first, having already identified enemy (4 and 5). Once the main player finds the enemy, he attacks and obtains the reward (6 and 7). Both agents then return to watching the dead area of the other until the next enemy appears. Three models, described below, were compared: the proposed method and two comparison targets. Baseline: DQN without communication. The main player learned standard DQN with the perspective that the player is viewing. Because the cameraman did not learn, this player continued to move randomly. Comm: DQN with communication, inspired by Commnet. The main player learns DQN with two perspectives: theirs and that of the cameraman. The communication vector is learned with a feedforward neural network. NaaA: The proposed method. The main player learned DQN with two perspectives: theirs and that of the cameraman. Transmissions of rewards and communications were performed using the proposed method. Training was performed over the course of 10 million steps. FIG3 Left demonstrates the proposed NaaA model outperformed the other two methods. Improvement was achieved by Adaptive DropConnect. It was confirmed that the cameraman observed the enemy through an episode, which could be interpreted as the cameraman reporting enemy positions. In addition to seeing the enemy, the cameraman observed the area behind the main player several times. This enabled the cameraman to observe enemy attacks while taking a better relative position. To further interpret this , a heatmap visualization of revenue earned by the agent is presented in FIG3 Right. The picture is a screen from Doom, recorded at the moment when the CNN filter was most activated. Figure 4 shows an example of learnt sequence of actions by our method. This paper proposed a NaaA model to address communication in MARL without a TTP based on two key ideas: inter-agent reward distribution and auction theory. Existing MARL communication methods have assumed the existence of a TTP, and hence could not be applied in peer-to-peer environments. The inter-agent reward distribution, making agents redistribute the rewards they received from the internal/external environment, was reviewed first. When an envy-free auction was introduced using auction theory, it was shown that agents would evaluate the counterfactual returns of other agents. The experimental demonstrated that NaaA outperformed a baseline method and a CommNet-based method. Furthermore, a Q-learning based algorithm, termed Adaptive DropConnect, was proposed to dynamically optimize neural network topology with counterfactual return evaluation as a further application. To evaluate this application, experiments were performed based on a single-agent platform, demonstrating that the proposed method produced improved experimental relative to existing methods. Future research may also be directed toward considering the connection between NaaA and neuroscience or neuroevolution. Edeleman propounded the concept of neural Darwinism BID5, in which group selection occurs in the brain. Inter-agent rewards, which were assumed in this paper, correspond to NTFs and could be used as a fitness function in genetic algorithms for neuroevolution such as hyperparameter tuning. As NaaA can be applied in peer-to-peer environments, the implementation of NaaA in blockchain BID24 is under consideration. This implementation would extend the areas where deep reinforcement learning could be applied. Bitcoin BID18 could be used for inter-agent reward distribution, and the auction mechanism could be implemented by smart contracts BID4. Using the NaaA reward design, it is hoped that the world may be united, allowing people to share their own representations on a global scale. The optimization problem in Eq:11 is made of two terms except of the constant, and the only second term is depends on b. Hence, we consider to optimize the second term. The optimal bidding priceŝ q t is given by the following equation. DISPLAYFORM0 From independence, the equation is solved if we solve the following problem. DISPLAYFORM1 Hence,b ijt can be derived as the solution which satisfies the following equation. DISPLAYFORM2 For simplicity, we let q = q jt and o = o ij,t+1. Then, the following equation holds. DISPLAYFORM3 DISPLAYFORM4

Represent the short summary for finding the corresponding scientific article (long)

Neuron as an Agent (NaaA) enable us to train multi-agent communication without a trusted third party.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Increasing model size when pretraining natural language representations often in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations, longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a , our best model establishes new state-of-the-art on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large. Full network pre-training (; ;) has led to a series of breakthroughs in language representation learning. Many nontrivial NLP tasks, including those that have limited training data, have greatly benefited from these pre-trained models. One of the most compelling signs of these breakthroughs is the evolution of machine performance on a reading comprehension task designed for middle and high-school English exams in China, the RACE test : the paper that originally describes the task and formulates the modeling challenge reports then state-of-the-art machine accuracy at 44.1%; the latest published reports their model performance at 83.2%; the work we present here pushes it even higher to 89.4%, a stunning 45.3% improvement that is mainly attributable to our current ability to build high-performance pretrained language representations. Evidence from these improvements reveals that a large network is of crucial importance for achieving state-of-the-art performance. It has become common practice to pre-train large models and distill them down to smaller ones for real applications. Given the importance of model size, we ask: Is having better NLP models as easy as having larger models? An obstacle to answering this question is the memory limitations of available hardware. Given that current state-of-the-art models often have hundreds of millions or even billions of parameters, it is easy to hit these limitations as we try to scale our models. Training speed can also be significantly hampered in distributed training, as the communication overhead is directly proportional to the number of parameters in the model. We also observe that simply growing the hidden size of a model such as BERT-large can lead to worse performance. Table 1 and Fig. 1 show a typical example, where we simply increase the hidden size of BERT-large to be 2x larger and get worse with this BERT-xlarge model. Model Hidden Size Parameters RACE (Accuracy) BERT-large 1024 334M 72.0% BERT-large (ours) 1024 334M 73.9% BERT-xlarge (ours) 2048 1270M 54.3% Table 1: Increasing hidden size of BERT-large leads to worse performance on RACE. Existing solutions to the aforementioned problems include model parallelization and clever memory management (; . These solutions address the memory limitation problem, but not the communication overhead and model degradation problem. In this paper, we address all of the aforementioned problems, by designing A Lite BERT (ALBERT) architecture that has significantly fewer parameters than a traditional BERT architecture. ALBERT incorporates two parameter reduction techniques that lift the major obstacles in scaling pre-trained models. The first one is a factorized embedding parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, we separate the size of the hidden layers from the size of vocabulary embedding. This separation makes it easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network. Both techniques significantly reduce the number of parameters for BERT without seriously hurting performance, thus improving parameter-efficiency. An ALBERT configuration similar to BERT-large has 18x fewer parameters and can be trained about 1.7x faster. The parameter reduction techniques also act as a form of regularization that stabilizes the training and helps with generalization. To further improve the performance of ALBERT, we also introduce a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed to address the ineffectiveness of the next sentence prediction (NSP) loss proposed in the original BERT. As a of these design decisions, we are able to scale up to much larger ALBERT configurations that still have fewer parameters than BERT-large but achieve significantly better performance. We establish new state-of-the-art on the well-known GLUE, SQuAD, and RACE benchmarks for natural language understanding. Specifically, we push the RACE accuracy to 89.4%, the GLUE benchmark to 89.4, and the F1 score of SQuAD 2.0 to 92.2. Learning representations of natural language has been shown to be useful for a wide range of NLP tasks and has been widely adopted (; ; ; ; ; 2019). One of the most significant changes in the last two years is the shift from pre-training word embeddings, whether standard or contextualized , to full-network pre-training followed by task-specific fine-tuning (; ; . In this line of work, it is often shown that larger model size improves performance. For example, show that across three selected natural language understanding tasks, using larger hidden size, more hidden layers, and more attention heads always leads to better performance. However, they stop at a hidden size of 1024. We show that, under the same setting, increasing the hidden size to 2048 leads to model degradation and hence worse performance. Therefore, scaling up representation learning for natural language is not as easy as simply increasing model size. In addition, it is difficult to experiment with large models due to computational constraints, especially in terms of GPU/TPU memory limitations. Given that current state-of-the-art models often have hundreds of millions or even billions of parameters, we can easily hit memory limits. To address this issue, propose a method called gradient checkpointing to reduce the memory requirement to be sublinear at the cost of an extra forward pass. propose a way to reconstruct each layer's activations from the next layer so that they do not need to store the intermediate activations. Both methods reduce the memory consumption at the cost of speed. In contrast, our parameter-reduction techniques reduce memory consumption and increase training speed. The idea of sharing parameters across layers has been previously explored with the Transformer architecture , but this prior work has focused on training for standard encoderdecoder tasks rather than the pretraining/finetuning setting. Different from our observations, show that networks with cross-layer parameter sharing (Universal Transformer, UT) get better performance on language modeling and subject-verb agreement than the standard transformer. Very recently, propose a Deep Equilibrium Model (DQE) for transformer networks and show that DQE can reach an equilibrium point for which the input embedding and the output embedding of a certain layer stay the same. Our observations show that our embeddings are oscillating rather than converging. combine a parameter-sharing transformer with the standard one, which further increases the number of parameters of the standard transformer. ALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text. Several researchers have experimented with pretraining objectives that similarly relate to discourse coherence. Coherence and cohesion in discourse have been widely studied and many phenomena have been identified that connect neighboring text segments (; ;). Most objectives found effective in practice are quite simple. Skipthought and FastSent sentence embeddings are learned by using an encoding of a sentence to predict words in neighboring sentences. Other objectives for sentence embedding learning include predicting future sentences rather than only neighbors and predicting explicit discourse markers . Our loss is most similar to the sentence ordering objective of , where sentence embeddings are learned in order to determine the ordering of two consecutive sentences. Unlike most of the above work, however, our loss is defined on textual segments rather than sentences. BERT uses a loss based on predicting whether the second segment in a pair has been swapped with a segment from another document. We compare to this loss in our experiments and find that sentence ordering is a more challenging pretraining task and more useful for certain downstream tasks. Concurrently to our work, also try to predict the order of two consecutive segments of text, but they combine it with the original next sentence prediction in a three-way classification task rather than empirically comparing the two. In this section, we present the design decisions for ALBERT and provide quantified comparisons against corresponding configurations of the original BERT architecture. The backbone of the ALBERT architecture is similar to BERT in that it uses a transformer encoder with GELU nonlinearities . We follow the BERT notation conventions and denote the vocabulary embedding size as E, the number of encoder layers as L, and the hidden size as H. , we set the feed-forward/filter size to be 4H and the number of attention heads to be H/64. There are three main contributions that ALBERT makes over the design choices of BERT. Factorized embedding parameterization. In BERT, as well as subsequent modeling improvements such as XLNet and RoBERTa, the WordPiece embedding size E is tied with the hidden layer size H, i.e., E ≡ H. This decision appears suboptimal for both modeling and practical reasons, as follows. From a modeling perspective, WordPiece embeddings are meant to learn context-independent representations, whereas hidden-layer embeddings are meant to learn context-dependent representations. As experiments with context length indicate, the power of BERT-like representations comes from the use of context to provide the signal for learning such context-dependent representations. As such, untying the WordPiece embedding size E from the hidden layer size H allows us to make a more efficient usage of the total model parameters as informed by modeling needs, which dictate that H E. From a practical perspective, natural language processing usually require the vocabulary size V to be large. 1 If E ≡ H, then increasing H increases the size of the embedding matrix, which has size V × E. This can easily in a model with billions of parameters, most of which are only updated sparsely during training. Therefore, for ALBERT we use a factorization of the embedding parameters, decomposing them into two smaller matrices. Instead of projecting the one-hot vectors directly into the hidden space of size H, we first project them into a lower dimensional embedding space of size E, and then project it to the hidden space. By using this decomposition, we reduce the embedding parameters from. This parameter reduction is significant when H E. We choose to use the same E for all word pieces because they are much more evenly distributed across documents compared to whole-word embedding, where having different embedding size (; ;) for different words is important. Cross-layer parameter sharing. For ALBERT, we propose cross-layer parameter sharing as another way to improve parameter efficiency. There are multiple ways to share parameters, e.g., only sharing feed-forward network (FFN) parameters across layers, or only sharing attention parameters. The default decision for ALBERT is to share all parameters across layers. All our experiments use this default decision unless otherwise specified We compare this design decision against other strategies in our experiments in Sec. 4.5. Similar strategies have been explored by (Universal Transformer, UT) and (Deep Equilibrium Models, DQE) for Transformer networks. Different from our observations, show that UT outperforms a vanilla Transformer. show that their DQEs reach an equilibrium point for which the input and output embedding of a certain layer stay the same. Our measurement on the L2 distances and cosine similarity show that our embeddings are oscillating rather than converging. Figure 2 shows the L2 distances and cosine similarity of the input and output embeddings for each layer, using BERT-large and ALBERT-large configurations (see Table 2). We observe that the transitions from layer to layer are much smoother for ALBERT than for BERT. These show that weight-sharing has an effect on stabilizing network parameters. Although there is a drop for both metrics compared to BERT, they nevertheless do not converge to 0 even after 24 layers. This shows that the solution space for ALBERT parameters is very different from the one found by DQE. Inter-sentence coherence loss. In addition to the masked language modeling (MLM) loss, BERT uses an additional loss called next-sentence prediction (NSP). NSP is a binary classification loss for predicting whether two segments appear consecutively in the original text, as follows: positive examples are created by taking consecutive segments from the training corpus; negative examples are created by pairing segments from different documents; positive and negative examples are sampled with equal probability. The NSP objective was designed to improve performance on downstream tasks, such as natural language inference, that require reasoning about the relationship between sentence pairs. However, subsequent studies found NSP's impact unreliable and decided to eliminate it, a decision supported by an improvement in downstream task performance across several tasks. We conjecture that the main reason behind NSP's ineffectiveness is its lack of difficulty as a task, as compared to MLM. As formulated, NSP conflates topic prediction and coherence prediction in a single task 2. However, topic prediction is easier to learn compared to coherence prediction, and also overlaps more with what is learned using the MLM loss. We maintain that inter-sentence modeling is an important aspect of language understanding, but we propose a loss based primarily on coherence. That is, for ALBERT, we use a sentence-order prediction (SOP) loss, which avoids topic prediction and instead focuses on modeling inter-sentence coherence. The SOP loss uses as positive examples the same technique as BERT (two consecutive segments from the same document), and as negative examples the same two consecutive segments but with their order swapped. This forces the model to learn finer-grained distinctions about discourse-level coherence properties. As we show in Sec. 4.6, it turns out that NSP cannot solve the SOP task at all (i.e., it ends up learning the easier topic-prediction signal, and performs at randombaseline level on the SOP task), while SOP can solve the NSP task to a reasonable degree, presumably based on analyzing misaligned coherence cues. As a , ALBERT models consistently improve downstream task performance for multi-sentence encoding tasks. We present the differences between BERT and ALBERT models with comparable hyperparameter settings in Table 2. Due to the design choices discussed above, ALBERT models have much smaller parameter size compared to corresponding BERT models. For example, ALBERT-large has about 18x fewer parameters compared to BERT-large, 18M versus 334M. If we set BERT to have an extra-large size with H = 2048, we end up with a model that has 1.27 billion parameters and under-performs (Fig. 1). In contrast, an ALBERT-xlarge configuration with H = 2048 has only 60M parameters, while an ALBERT-xxlarge configuration with H = BERT base 108M 12 768 768 False large 334M 24 1024 1024 False xlarge 1270M 24 2048 2048 False ALBERT base 12M 12 768 128 True large 18M 24 1024 128 True xlarge 60M 24 2048 128 True xxlarge 235M 12 4096 128 True Table 2: The configurations of the main BERT and ALBERT models analyzed in this paper. 4096 has 233M parameters, i.e., around 70% of BERT-large's parameters. Note that for ALBERTxxlarge, we mainly report on a 12-layer network because a 24-layer network (with the same configuration) obtains similar but is computationally more expensive. This improvement in parameter efficiency is the most important advantage of ALBERT's design choices. Before we can quantify this advantage, we need to introduce our experimental setup in more detail. To keep the comparison as meaningful as possible, we follow the BERT setup in using the BOOKCORPUS and English Wikipedia for pretraining baseline models. These two corpora consist of around 16GB of uncompressed text. We format our inputs as ", where x 1 = x 1,1, x 1,2 · · · and x 2 = x 1,1, x 1,2 · · · are two segments. We always limit the maximum input length to 512, and randomly generate input sequences shorter than 512 with a probability of 10%. Like BERT, we use a vocabulary size of 30,000, tokenized using SentencePiece as in XLNet. We generate masked inputs for the MLM targets using n-gram masking, with the length of each n-gram mask selected randomly. The probability for the length n is given by p(n) = 1/n N k=1 1/k We set the maximum length of n-gram (i.e., n) to be 3 (i.e., the MLM target can consist of up to a 3-gram of complete words, such as "White House correspondents"). All the model updates use a batch size of 4096 and a LAMB optimizer with learning rate 0.00176 . We train all models for 125,000 steps unless otherwise specified. Training was done on Cloud TPU V3. The number of TPUs used for training ranged from 64 to 1024, depending on model size. The experimental setup described in this section is used for all of our own versions of BERT as well as ALBERT models, unless otherwise specified. To monitor the training progress, we create a development set based on the development sets from SQuAD and RACE using the same procedure as in Sec. 4.1. We report accuracies for both MLM and sentence classification tasks. Note that we only use this set to check how the model is converging; it has not been used in a way that would affect the performance of any downstream evaluation, such as via model selection. and, we evaluate our models on three popular benchmarks: The General Language Understanding Evaluation (GLUE) benchmark , two versions of the Stanford Question Answering Dataset (SQuAD;, and the ReAding Comprehension from Examinations (RACE) dataset . For completeness, we provide description of these benchmarks in Appendix A.1. As in, we perform early stopping on the development sets, on which we report all comparisons except for our final comparisons based on the task leaderboards, for which we also report test set . We are now ready to quantify the impact of the design choices described in Sec. 3, specifically the ones around parameter efficiency. The improvement in parameter efficiency showcases the most important advantage of ALBERT's design choices, as shown in Table 3: with only around 70% of BERT-large's parameters, ALBERT-xxlarge achieves significant improvements over BERT-large, as measured by the difference on development set scores for several representative downstream tasks: SQuAD v1.1 (+1.9%), SQuAD v2.0 (+3.1%), MNLI (+1.4%), SST-2 (+2.2%), and RACE (+8.4%). We also observe that BERT-xlarge gets significantly worse than BERT-base on all metrics. This indicates that a model like BERT-xlarge is more difficult to train than those that have smaller parameter sizes. Another interesting observation is the speed of data throughput at training time under the same training configuration (same number of TPUs). Because of less communication and fewer computations, ALBERT models have higher data throughput compared to their corresponding BERT models. The slowest one is the BERT-xlarge model, which we use as a baseline. As the models get larger, the differences between BERT and ALBERT models become bigger, e.g., ALBERT-xlarge can be trained 2.4x faster than BERT-xlarge. Table 3: Dev set for models pretrained over BOOKCORPUS and Wikipedia for 125k steps. Here and everywhere else, the Avg column is computed by averaging the scores of the downstream tasks to its left (the two numbers of F1 and EM for each SQuAD are first averaged). Next, we perform ablation experiments that quantify the individual contribution of each of the design choices for ALBERT. Table 4 shows the effect of changing the vocabulary embedding size E using an ALBERT-base configuration setting (see Table 2), using the same set of representative downstream tasks. Under the non-shared condition (BERT-style), larger embedding sizes give better performance, but not by much. Under the all-shared condition (ALBERT-style), an embedding of size 128 appears to be the best. Based on these , we use an embedding size E = 128 in all future settings, as a necessary step to do further scaling. Table 5 presents experiments for various cross-layer parameter-sharing strategies, using an ALBERT-base configuration (Table 2) with two embedding sizes (E = 768 and E = 128). We compare the all-shared strategy (ALBERT-style), the not-shared strategy (BERT-style), and intermediate strategies in which only the attention parameters are shared (but not the FNN ones) or only the FFN parameters are shared (but not the attention ones). Table 4: The effect of vocabulary embedding size on the performance of ALBERT-base. The all-shared strategy hurts performance under both conditions, but it is less severe for E = 128 (-1.5 on Avg) compared to E = 768 (-2.5 on Avg). In addition, most of the performance drop appears to come from sharing the FFN-layer parameters, while sharing the attention parameters in no drop when E = 128 (+0.1 on Avg), and a slight drop when E = 768 (-0.7 on Avg). There are other strategies of sharing the parameters cross layers. For example, We can divide the L layers into N groups of size M, and each size-M group shares parameters. Overall, our experimental shows that the smaller the group size M is, the better the performance we get. However, decreasing group size M also dramatically increase the number of overall parameters. We choose all-shared strategy as our default choice. Table 5: The effect of cross-layer parameter-sharing strategies, ALBERT-base configuration. We compare head-to-head three experimental conditions for the additional inter-sentence loss: none (XLNet-and RoBERTa-style), NSP (BERT-style), and SOP (ALBERT-style), using an ALBERTbase configuration. Results are shown in The on the intrinsic tasks reveal that the NSP loss brings no discriminative power to the SOP task (52.0% accuracy, similar to the random-guess performance for the "None" condition). This allows us to conclude that NSP ends up modeling only topic shift. In contrast, the SOP loss does solve the NSP task relatively well (78.9% accuracy), and the SOP task even better (86.5% accuracy). Even more importantly, the SOP loss appears to consistently improve downstream task performance for multi-sentence encoding tasks (around +1% for SQuAD1.1, +2% for SQuAD2.0, +1.7% for RACE), for an Avg score improvement of around +1%. In this section, we check how depth (number of layers) and width (hidden size) affect the performance of ALBERT. Table 7 shows the performance of an ALBERT-large configuration (see Table 2) using different numbers of layers. Networks with 3 or more layers are trained by fine-tuning using the parameters from the depth before (e.g., the 12-layer network parameters are fine-tuned from the checkpoint of the 6-layer network parameters). 4 Similar technique has been used in. If we compare a 3-layer ALBERT model with a 1-layer ALBERT model, although they have the same number of parameters, the performance increases significantly. However, there are diminishing returns when continuing to increase the number of layers: the of a 12-layer network are relatively close to the of a 24-layer network, and the performance of a 48-layer network appears to decline. Table 7: The effect of increasing the number of layers for an ALBERT-large configuration. A similar phenomenon, this time for width, can be seen in Table 8 for a 3-layer ALBERT-large configuration. As we increase the hidden size, we get an increase in performance with diminishing returns. At a hidden size of 6144, the performance appears to decline significantly. We note that none of these models appear to overfit the training data, and they all have higher training and development loss compared to the best-performing ALBERT configurations. Table 8: The effect of increasing the hidden-layer size for an ALBERT-large 3-layer configuration. The speed-up in Table 3 indicate that data-throughput for BERT-large is about 3.17x higher compared to ALBERT-xxlarge. Since longer training usually leads to better performance, we perform a comparison in which, instead of controlling for data throughput (number of training steps), we control for the actual training time (i.e., let the models train for the same number of hours). In Table 9, we compare the performance of a BERT-large model after 400k training steps (after 34h of training), roughly equivalent with the amount of time needed to train an ALBERT-xxlarge model with 125k training steps (32h of training). Table 9: The effect of controlling for training time, BERT-large vs ALBERT-xxlarge configurations. After training for roughly the same amount of time, ALBERT-xxlarge is significantly better than BERT-large: +1.5% better on Avg, with the difference on RACE as high as +5.2%. In Section 4.7, we show that for ALBERT-large (H=1024), the difference between a 12-layer and a 24-layer configuration is small. Does this still hold for much wider ALBERT configurations, such as ALBERT-xxlarge (H=4096)? Number of layers SQuAD1. Table 10: The effect of a deeper network using an ALBERT-xxlarge configuration. The answer is given by the from Table 10. The difference between 12-layer and 24-layer ALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than a 12-layer configuration. The experiments done up to this point use only the Wikipedia and BOOKCORPUS datasets, as in. In this section, we report measurements on the impact of the additional data used by both XLNet and RoBERTa. Fig. 3a plots the dev set MLM accuracy under two conditions, without and with additional data, with the latter condition giving a significant boost. We also observe performance improvements on the downstream tasks in Table 11, except for the SQuAD benchmarks (which are Wikipedia-based, and therefore are negatively affected by out-of-domain training material). Table 11: The effect of additional training data using the ALBERT-base configuration. We also note that, even after training for 1M steps, our largest models still do not overfit to their training data. As a , we decide to remove dropout to further increase our model capacity. The plot in Fig. 3b shows that removing dropout significantly improves MLM accuracy. Intermediate evaluation on ALBERT-xxlarge at around 1M training steps (Table 12) also confirms that removing dropout helps the downstream tasks. There is empirical and theoretical evidence showing that a combination of batch normalization and dropout in Convolutional Neural Networks may have harmful . To the best of our knowledge, we are the first to show that dropout can hurt performance in large Transformer-based models. However, the underlying network structure of ALBERT is a special case of the transformer and further experimentation is needed to see if this phenomenon appears with other transformer-based architectures or not. Table 12: The effect of removing dropout, measured for an ALBERT-xxlarge configuration. The we report in this section make use of the training data used by, as well as the additional data used by and. We report state-of-the-art under two settings for fine-tuning: single-model and ensembles. In both settings, we only do single-task fine-tuning. Following, on the development set we report the median over five runs. The single-model ALBERT configuration incorporates the best-performing settings discussed: an ALBERT-xxlarge configuration (Table 2) using combined MLM and SOP losses, and no dropout. The checkpoints that contribute to the final ensemble model are selected based on development set performance; the number of checkpoints considered for this selection range from 6 to 17, depending on the task. For the GLUE (Table 13) and RACE (Table 14) benchmarks, we average the model predictions for the ensemble models, where the candidates are fine-tuned from different training steps using the 12-layer and 24-layer architectures. For SQuAD (Table 14), we average the prediction scores for those spans that have multiple probabilities; we also average the scores of the "unanswerable" decision. Both single-model and ensemble indicate that ALBERT improves the state-of-the-art significantly for all three benchmarks, achieving a GLUE score of 89.4, a SQuAD 2.0 test F1 score of 92.2, and a RACE test accuracy of 89.4. The latter appears to be a particularly strong improvement, a jump of +17.4% absolute points over BERT, +7.6% over XLNet, +6.2% over RoBERTa, and 5.3% over DCMI+, an ensemble of multiple models specifically designed for reading comprehension tasks. Our single model achieves an accuracy of 86.5%, which is still 2.4% better than the state-of-the-art ensemble model. Table 13: State-of-the-art on the GLUE benchmark. For single-task single-model , we report ALBERT at 1M steps (comparable to RoBERTa) and at 1.5M steps. The ALBERT ensemble uses models trained with 1M, 1.5M, and other numbers of steps. While ALBERT-xxlarge has less parameters than BERT-large and gets significantly better , it is computationally more expensive due to its larger structure. An important next step is thus to speed up the training and inference speed of ALBERT through methods like sparse attention and block attention . An orthogonal line of research, which could provide additional representation power, includes hard example mining efficient language modeling training. Additionally, although we have convincing evidence that sentence order prediction is a more consistently-useful learning task that leads to better language representations, we hypothesize that there could be more dimensions not yet captured by the current self-supervised training losses that could create additional representation power for the ing representations. RACE RACE is a large-scale dataset for multi-choice reading comprehension, collected from English examinations in China with nearly 100,000 questions. Each instance in RACE has 4 candidate answers. Following prior work, we use the concatenation of the passage, question, and each candidate answer as the input to models. Then, we use the representations from the "[CLS]" token for predicting the probability of each answer. The dataset consists of two domains: middle school and high school. We train our models on both domains and report accuracies on both the development set and test set. A.2 HYPERPARAMETERS Hyperparameters for downstream tasks are shown in Table 15. We adapt these hyperparameters from,, and Yang et al. (2019

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A new pretraining method that establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large.

Represent the short summary for finding the corresponding scientific article (long).

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the scientific article to find a short TLDR

Question: Structured tabular data is the most commonly used form of data in industry according to a Kaggle ML and DS Survey. Gradient Boosting Trees, Support Vector Machine, Random Forest, and Logistic Regression are typically used for classification tasks on tabular data. The recent work of Super Characters method using two-dimensional word embeddings achieved state-of-the-art in text classification tasks, showcasing the promise of this new approach. In this paper, we propose the SuperTML method, which borrows the idea of Super Characters method and two-dimensional embeddings to address the problem of classification on tabular data. For each input of tabular data, the features are first projected into two-dimensional embeddings like an image, and then this image is fed into fine-tuned ImageNet CNN models for classification. Experimental have shown that the proposed SuperTML method have achieved state-of-the-art on both large and small datasets. In data science, data is categorized into structured data and unstructured data. Structured data is also known as tabular data, and the terms will be used interchangeably. Anthony Goldbloom, the founder and CEO of Kaggle observed that winning techniques have been divided by whether the data was structured or unstructured BID12. Currently, DNN models are widely applied for usage on unstructured data such as image, speech, and text. According to Anthony, "When the data is unstructured, its definitely CNNs and RNNs that are carrying the day" BID12. The successful CNN model in the ImageNet competition BID8 has outperformed human for image classification task by ResNet BID6 since 2015.On the other side of the spectrum, machine learning models such as Support Vector Machine (SVM), Gradient Boosting Trees (GBT), Random Forest, and Logistic Regression, have been used to process structured data. According to a recent survey of 14,000 data scientists by , a subdivision of structured data known as relational data is reported as the most popular type of data in industry, with at least 65% working daily with relational data. Regarding structured data competitions, Anthony says that currently XGBoost is winning practically every competition in the structured data category BID4. XGBoost BID2 is one popular package implementing the Gradient Boosting method. Recent research has tried using one-dimensional embedding and implementing RNNs or one-dimensional CNNs to address the TML (Tabular Machine Learning) tasks, or tasks that deal with structured data processing BID7 BID11, and also categorical embedding for tabular data with categorical features BID5. However, this reliance upon onedimensional embeddings may soon come to change. Recent NLP research has shown that the two-dimensional embedding of the Super Characters method BID9 is capable of achieving state-of-the-art on large dataset benchmarks. The Super Characters method is a two-step method that was initially designed for text classification problems. In the first step, the characters of the input text are drawn onto a blank image. In the second step, the image is fed into two-dimensional CNN models for classification. The two-dimensional CNN models are trained by fine-tuning from pretrained models on large image dataset, e.g. ImageNet. In this paper, we propose the SuperTML method, which borrows the concept of the Super Characters method to address TML problems. For each input, tabular features are first projected onto a two-dimensional embedding and fed into fine-tuned two-dimensional CNN models for classification. The proposed SuperTML method handles the categorical type and missing values in tabular data automatically, without need for explicit conversion into numerical type values. The SuperTML method is motivated by the analogy between TML problems and text classification tasks. For any sample given in tabular form, if its features are treated like stringified tokens of data, then each sample can be represented as a concatenation of tokenized features. By applying this paradigm of a tabular sample, the existing CNN models used in Super Characters method could be extended to be applicable to TML problems. As mentioned in the introduction, the combination of twodimensional embedding (a core competency of the Super Characters methodology) and pre-trained CNN models has achieved state-of-the-art on text classification tasks. However, unlike the text classification problems studied in BID9, tabular data has features in separate dimensions. Hence, generated images of tabular data should reserve some gap between features in different dimensions in order to guarantee that features will not overlap in the generated image. SuperTML is composed of two steps, the first of which is two-dimensional embedding. This step projects features in the tabular data onto the generated images, which will be called the SuperTML images in this paper. The conversion of tabular training data to SuperTML image is illustrated in Figure 1, where a collection of samples containing four tabular features is being sorted. The second step is using pretrained CNN models to finetune on the generated SuperTML images. Figure 1 only shows the generation of SuperTML images for the training data. It should be noted that for inference, each instance of testing data goes through the same preprocessing to generate a SuperTML image (all of which use the same configuration of two-dimensional embedding) before getting fed into the CNN classification model. the generated SuperTML images. 9: return the trained CNN model on the tabular data known as SuperTML VF, is described in Algorithm 1.To make the SuperTML more autonomous and remove the dependency on feature importance calculation done in Algorithm 1, the SuperTML EF method is introduced in Algorithm 2. It allocates the same size to every feature, and thus tabular data can be directly embedded into SuperTML images without the need for calculating feature importance. This algorithm shows even better than 1, which will be described more in depth later in the experimental section. The data statistics from UCI Machine Learning Repository is shown in TAB2. " This is perhaps the best known database to be found in the pattern recognition literature"1. The Iris dataset is widely used in machine learning courses and tutorials. FIG2 shows an example of a generated SuperTML image, created using Iris data. The experimental of using SEnet-154 shown in Table 2 for each feature of the sample do 3:Draw the feature in the same font size without overlapping, such that the total features of the sample will occupy the image size as much as possible. For this dataset 2, we use SuperTML VF, which gives features different sizes on the SupterTML image according to their importance score. The feature importance score is obtained using the XGBoost package BID2. One example of a SuperTML image created using data from this dataset is shown in FIG2. The in Table 2 shows that the SuperTML method obtained a slightly better accuracy than XGBoost on this dataset. The task of this Adult dataset 3 is to predict whether a persons income is larger or smaller than 50,000 dollars per year based on a collection of surveyed data. For categorical features that are represented by strings, the Squared English Word (SEW) method BID10 Figure 3. SuperTML VF image example from Adult dataset. This sample has age = 59, capital gain = 0, capital loss = 0, hours per week = 40, fnlweight = 372020, education number = 13, occupation = "?" (missing value), marital status = "Married-civ-spouse", relationship = "Husband", workclass = "?" (missing value), education = "Bachelors", sex = "Male", race = "White", native country = "United-States".is used. One example of a generated SuperTML image is given in Figure 3. Table 2 shows the on Adult dataset. We can see that on this dataset, the SuperTML method still has a higher accuracy than the fine-tuned XGBoost model, outperforming it by 0.32% points of accuracy. The Higgs Boson Machine Learning Challenge involved a binary classification task to classify quantum events as signal or . It was hosted by Kaggle, and though the contest is over, the challenge data is available on opendata BID1. It has 25,000 training samples, and 55,000 testing samples. Each example has 30 features, each of which is stored as a real number value. In this challenge, AMS score BID0 (a) SuperTML EF event example. is used as the performance metric. FIG4 shows two examples of generated SuperTML images. TAB3 shows the comparison of different algorithms. The DNN method and XGBoost used in the first two rows are using the numerical values of the features as input to the models, which is different from the SuperTML method of using two-dimensional embeddings. It shows that SuperTML EF method gives the best AMS score of 3.979. In addition, the SuperTML EF gives better than SuperTME VF , which indicates SuperTML method can work well without the calculation of the importance scores. The proposed SuperTML method borrows the idea of twodimensional embedding from Super Characters and transfers the knowledge learned from computer vision to the structured tabular data. Experimental shows that the proposed SuperTML method has achieved state-of-the-art on both large and small tabular dataset.

Represent the short summary for finding the corresponding scientific article (long)!

Deep learning on structured tabular data using two-dimensional word embedding with fine-tuned ImageNet pre-trained CNN model.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.