Question: Represent the article for finding a single-sentence TLDR

Question: Using Recurrent Neural Networks (RNNs) in sequence modeling tasks is promising in delivering high-quality but challenging to meet stringent latency requirements because of the memory-bound execution pattern of RNNs. We propose a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference. Leveraging the error-resilient feature of nonlinear activation functions used in RNNs, we propose to use a lightweight little module that approximates the original RNN layer, which is referred to as the big module, to compute activations of the insensitive region that are more error-resilient. The expensive memory access and computation of the big module can be reduced as the are only used in the sensitive region. Our method can reduce the overall memory access by 40% on average and achieve 1.54x to 1.75x speedup on CPU-based server platform with negligible impact on model quality. Recurrent Neural Networks (RNNs) play a critical role in many natural language processing (NLP) tasks, such as machine translation ), speech recognition , and speech synthesis, owing to the capability of modeling sequential data. These RNN-based services deployed in both data-center and edge devices often process inputs in a streaming fashion, which demands a real-time interaction. For instance, in cloud-based translation tasks, multiple requests need to be served with very stringent latency limit, where inference runs concurrently and individually . For on-device speech recognition as an automated assistant, latency is the primary concern to pursue a fast response . However, serving RNN-based models in latency-sensitive scenarios is challenging due to the low data reuse, and thus low resource utilization as memory-bound General Matrix-Vector multiplication (GEMV) is the core compute pattern of RNNs. Accessing weight matrix from off-chip memory is the bottleneck of GEMV-based RNN execution as the weight data almost always cannot fit in on-chip memory. Moreover, accessing weights repeatedly at each time-step, especially in sequenceto-sequence models, makes the memory-bound problem severer. Subsequently, the on-chip computing resources would be under-utilized. Although batching is a walk-around for low-utilization, using a large batch size is not favored in latency-sensitive scenarios such as speech recognition and translation. In essence, the RNN inference is not a simple GEMV. With non-linearity followed the GEMV operation as the activation functions, the RNN inference operation is "activated" GEMV. These nonlinear activation functions as used in neural networks bring error resilience. As shown in Figure 1, sigmoid and tanh functions in Gated RNNs such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have insensitive regionsgreen shaded regions -where the outputs are saturated and resilient to errors in pre-activation accumulated . In other words, not all computations in RNNs need to be accurate. Can we leverage this error resilience in RNNs to reduce the memory access and eventually achieve speedup? To this end, we propose a big-little dual-module inference that regarding the original RNN layer as the big module, and use a parameterized little module to approximate the big module to help reduce redundant weight accesses. The philosophy of dual-module inference is using approximated computed by the memory-efficient little module in the insensitive region, and using accurate computed by the memory-intensive big module in the sensitive region. For this reason, the final outputs are the mixture of the big-little module. With the memory-efficient little module computes for the insensitive region, we can reduce the expensive data access and computation of the big module and thus reduce overall memory access and computation cost. The (in)sensitive region is dynamically determined using the little module . Because of the error resilience, using approximated in the insensitive region has a negligible impact on the overall model quality but creates a significant acceleration potential. Given the trade-off between accuracy and efficiency, the little module needs to be sufficiently accurate while being as much lightweight as possible. To achieve this, we first use a dimension reduction method -random projection -to reduce the parameter size of the little module and thus reducing data accesses. Then, we quantize the weights of the little module to lower the overhead further. Because we only need the little module outputs in the insensitive region that is error-resilient, we can afford aggressively low bit-width. Compared with common sparsification schemes, our hybrid approach avoids indexing overheads and therefore successfully achieves practical speedup. We evaluate our method on language modeling and neural machine translation using RNN-based models and measure the performance, i.e., wall-clock execution time, on CPU-based server platform. With overall memory access data reduced by 40% on average, our method can achieve 1.54x to 1.75x speedup with negligible impact on model quality. In this section, we discuss the error resilience of RNNs. As shown in Fig. 1, the nonlinear activation functions -sigmoid and tanh -have insensitive regions where the output activations are resilient to errors introduced in pre-activation accumulation . We take a single LSTM layer for language modeling over PTB dataset as an illustrative example. The baseline perplexity (PPL) is 80.64. We consider two cases: adding a random error vector under norm distribution into the pre-activation accumulation in the sensitive regions of four gates; adding errors to the insensitive regions. We separate the (in)sensitive regions by 50% based on the activation magnitude. As listed in Table 1, we report the PPL on the testing set and the average cosine similarity between the activations of the baseline model and the error-introduced model. Before applying the nonlinear activation functions, the cosine similarity of two cases -adding errors in the sensitive region or the insensitive region -are in the same level. However, we observe that after the nonlinear gates, the cosine similarity in the insensitive case is much closer to one (i.e., fewer output errors) than that in the sensitive case. We further compare the PPL of these two cases, and we observe that introducing errors in the insensitive region causes little quality degradation. The selection of which neurons should be in the (in)sensitive region is dynamic and input-dependent, which can be seen in Figure 2. Unlike the static weight sparsity that we can prune the unused connections offline in advance, the dynamic region speculation requires a very lightweight criterion for real-time processing. Taking all these into account, we propose a dual-model inference method that efficiently determines (in)sensitive region and significantly saves the memory access and computational cost. Firstly, we explain the dual-module inference by taking a fully-connected (FC) layer as an example and then extend it to LSTM and GRU. For an FC layer with unit batch size, the operation is typically formulated as a = ϕ(y), y = W x + b, where W is a weight matrix (W ∈ R n×d), x is an input vector (x ∈ R d), b is a bias vector (b ∈ R n), a is an activated output vector (a ∈ R n), and ϕ is an activation function. The core computation is matrix-vector multiplication (GEMV), i.e., W x. Both the amount of computation and memory access are O(nd); therefore, it is memory-bounded since the operation intensity is O according to the Roofline model analysis . Accessing weights from the off-chip memory is the bottleneck in terms of both the latency and energy. Our work aims at reducing the memory access of weight matrices for GEMV-based RNN inference. We show in Section 2 that not all values in y need accurate computation, and those that belong to the insensitive region can afford some level of approximation. In other words, we only need accurate computation and expensive memory access in the sensitive region of y and skip computation and memory access to weights that contribute to the insensitive region of y. With that, we still need approximated in the insensitive region. Therefore, we propose to learn a lightweight little module from the original trained layer, here we refer the original layer as the big module. Essentially, our little module is executed in a low-dimensional and low-precision space, thus termed as LL module; by contrast, the original big module with high dimension and high precision is called HH module. Let the outputs from these two modules be y LL and y HH, respectively. If the LL module approximates the HH module well, the final output vector -a mixture of from the HH and the LL modules -can be assembled by where m ∈ {0, 1} n is a binary mask vector for the output switching. m i equals 1 in the sensitive region while it switches to 0 in the insensitive region. The overall saving comes from skipping memory access to the big module while paying the overhead of accessing and computing of the little module. As the HH module is the original pre-trained layer, we only need to construct the LL module. Delivering a lightweight little module at inference time is crucial to achieving real wall-clock time speedup. As discussed earlier, the sparsification method usually suffers from severe indexing overheads; therefore, we turn to other approaches. In this work, we propose a hybrid compression with dimension reduction and data quantization to keep the little module as efficient as possible in computation and storage. The low dimension and low precision give birth to the desired LL module. We emphasize two objects that should be reached in the design of LL module: much lower computation and memory overheads than the HH module; approximating the outputs of HH module accurately. First, we introduce sparse random projection to reduce the dimension of x from R d to R k where k d. Subsequently, the parameter size of the LL module is O(nk), which is much smaller compared with the parameter size O(nd) of the HH module. Random projection is a common technique for dimension reduction that preserves distances in Euclidean space (; ; ;). The dimension reduction step can be formulated as where P is a sparse random matrix (P ∈ k×d, the probability of P ij being −1, 0, and 1 is 1 6, 2 3, and 1 6, respectively). Note that k is configurable according to actual needs to balance the accuracy loss and inference cost. We choose the value of k according to: where n is the number of rows in W and is a real number in. Second, after the dimension reduction, we quickly construct a lightweight little module in the lowdimensional space to approximate the pre-trained big module. The parameters of the latter (i.e., W HH and b HH) are kept frozen while the parameters of the former (i.e., W LL and b LL) are updated by stochastic gradient descent (SGD) to minimize the following loss function: where S is the mini-batch size. Essentially, for each pair of big-little modules, we apply linear regression on the little module to approximate the function of the big module and optimize the mean square error of the two. Apparently, the parameter size of W LL is O(nk), much smaller than the original weight W HH of O(nd) in the high-dimensional space. Even if further considering the projection cost of O(kd), the overhead is still much lower than the vanilla inference. In this way, the memory-bound issue in GEMV-based models can be greatly alleviated; the computational complexity is also reduced. The SGD overhead for constructing the above module can be amortized by the pattern of "construct-once-inference-forever". Finally, based on the constructed low-dimensional module, we also apply data quantization technique to reduce the parameter precision. Data quantization can further shrink the storage space of LL parameters due to the shorter bit-width. The input x is also quantized during run-time to reduce the computation cost. In our design, we apply one-time uniform quantization on W LL to avoid complicated calculations. Although some other accurate quantization methods are available as well, we find that one-time quantization works well in our dual-module inference given in Equation. This error tolerance is benefit from the fact that the computation in the insensitive region has a small influence on the final outputs. The dual-module inference relies on a binary mask m to switch between outputs of the "accurate & costly" HH module and the "approximated & efficient" LL module. Hence, the generation of m is a crucial factor to control the overall performance by adjusting the trade-off between accuracy and efficiency. Thanks to the saturation region of the nonlinear activation functions in RNNs, such as sigmoid and tanh, we observe a unipolar or bipolar distribution of their outputs, as depicted in Figure 3. This affords two excellent opportunities: It is possible to remove the majority of the computation and access from the costly HH module by setting the peak areas in Figure 3 as insensitive regions; The saturation output values in those regions such as near 0 in sigmoid and near ±1 in tanh additionally allow inaccurate computations because the outputs are insensitive to approximated values. According to the above observations and analysis, we design a specific criterion for each activation function. In particular, they are governed by where θ sigmoid > 0, θ − tanh < 0, and θ + tanh > 0 are constant thresholds. Note that these thresholds can be searched to a target insensitive ratio using validation dataset or be tuned at run-time that acts as a knob for accuracy-efficiency trade-off. The overall implementation is provided in Algorithm 1. After the construction of the LL model, the consequent dual-module inference needs five steps: Dimension reduction and data quantization for each dynamical input x as x Step 1: Step 2: Step 3: Generating m according to Equation (We discuss how to apply the proposed dual-module inference for an FC layer to RNNs, including LSTM and GRU. We will explain the LSTM implementation for illustration, while the extension to GRU is quite straightforward. The dynamics of an LSTM cell can be described as where f, i, o are the states of forget, input, and output gate, respectively, and g is the input activation. Each of them has its own bias vector and weight matrices. c and h are the cellular and hidden states of the hidden layer, respectively. σ(·) and θ(·) are sigmoid function and tanh function, respectively. The computation of each gate is similar to an FC-like layer; therefore, Algorithm 1 still holds. The first difference is the two GEMV computations in each gate; we apply dimension reduction, construction of the LL module, and data quantization on both GEMV computations. The second difference is that there is an additional temporal dimension in RNNs. We should guarantee the approximation performance of the LL module at all time steps. Taking the forget gate as an example, the linear map works for both x LL (t) = P x x HH (t) and h LL (t − 1) = P h h HH (t − 1). The loss function for constructing the LL module is slightly modified to Here the minimization considers not only S training samples in each mini-batch but also T time steps. The data quantization, switching mask (i.e., m) generation, and output assembling is the same as Algorithm 1 describes. Applying to other gates is similar so we do not discuss them to avoid repetition. Note that the input x and hidden state h can have different sizes, termed as d x and d h, respectively. For simplicity, we set P x ∈ R k×dx and P h ∈ R k×d h to let x LL and h LL to the same length k. For the g gate with tanh function, we set |θ − tanh | = |θ + tanh | also for simplicity; however, different magnitudes are allowed. The target of our dual-module inference method is to reduce the expensive off-chip memory access of the big module with the help of the little module. We introduce an insensitive ratio as the number of outputs using the little module over entire outputs. The ratio can be interpreted as the zero ratio in mask m as in Equation 1. In other words, the higher insensitive ratio will have less memory access to the big module. For example, obtaining a ratio of 50% in reducing 50% of weight matrix accessing in a GEMV operation. The choice of accurate ratio determines the model inference quality, and it is a knob to trade-off model inference quality vs. latency at run-time. The overhead of dual-module inference is small due to the use of dimension reduction and quantization. When choosing reduced dimension k and low-precision bit-width of the little module, we use Equation 3 with = 0.5 and INT8 quantization by default. We also explore different levels of dimension reduction and quantization in Section 4.3 and Section 4.4. As shown in Figure 4, we compare memory access data and operations between the single-module -the base case -and the little module of dual-module inference using a set of LSTM and GRU layers. On average, the little module accounts 10% storage overhead and 40% operation overhead compared with the base case. Note that we count the number of operations in Figure 4 regardless of precision; and the little module computation overhead can be further reduced using low-precision compute kernel as we used in performance evaluation. We first evaluate the model inference quality and execution time under different insensitive ratio and then conduct two sensitivity studies on dimension reduction and quantization. Our method is evaluated on CPU-based server platform (Intel(R) Xeon(R) CPU E5-2698 v4) as most inference workloads run on CPUs . We use PyTorch to train the little module and evaluate inference quality. The baseline implementation is the PyTorch CPU version with Intel MKL (version 2019.4) as the back-end BLAS kernel library. Our custom kernel implementation uses a multi-threaded MKL dot-product kernel at BLAS level-1 to compute the big module instead of BLAS level-2 or level-3 kernels. The kernel-wise performance is measured as wall-clock time and averaged with 1000 runs, assuming cold cache at the execution of each RNN cell representing the real-world cases, for example in the decoder of seq2seq model. We first evaluate our method on single-layer LSTM & GRU used in language modeling tasks and then on multi-layer stacked LSTM in GNMT model used in machine translation tasks -a standard benchmark model for inference as in MLPerf 1. We train the little module while freezing the parameters of the big module, and we use the same training set and validation set to run SGD optimization. We first evaluate our method on single-layer LSTMs/GPUs. Our implementations are adapted from the word-level language modeling example from PyTorch with same hyper-parameters to train baseline models. We report word-level perplexity (PPL) as the measure of model quality. As listed in Table 2, the baseline LSTM model achieves 80.64 PPL at the latency of 1.477ms. Then, we varying the insensitive ratio to show the quality-performance trade-off; the larger insensitive ratio indicates more are from the little module and less memory access to compute the big module. As we increase the insensitive ratio, we observe the degradation of quality as the perplexity increases during a gradual reduction in execution time. When the insensitive ratio is 50%, the perplexity is slightly increased to 81.36, which is negligible in language modeling tasks, while the inference speedup is 1.67x. We observe a similar quality-performance trade-off for LSTM with 750 hidden units. Comparing the case of base LSTM with 750 hidden units with dual-module LSTM with 1500 hidden units and 50% insensitive ratio, although the memory access reduction is at the same level, our proposed dualmodule approach achieves much better model quality because we kept the expressive power of a larger LSTM layer. We further report the using single-layer GRU on word-level language modeling tasks as in Table 3. Using dual-module inference on GRUs expresses the similar quality-performance trade-off as of LSTMs. Our dual-module method is generally applicable to both LSTMs and GRUs. Given the promising on language modeling, we further investigate Neural Machine Translation (NMT), which is a promising end-to-end learning approach for automated translation . The base model 2 consists of a four-layer stacked LSTM in both the encoder and the decoder of the sequence-to-sequence modeling. We focus on the speedup of the decoder since it is the most memory intensive and the most time-consuming part (95%). The decoder has a four-layer unidirectional LSTM with hidden size 1024 with residual connections starting from the third layer, i.e., the input size of the third and fourth layer is 2048. Our experiments show de-tokenized BLEU score to measure the model inference quality on the public WMT16 English-German dataset. The baseline model obtains a BLEU score of 24.32. We replace the LSTM layers in the decoder with our proposed dual-module-based LSTM layers. Similar to single-layer LSTM , using the little module computed in the insensitive region can reduce overall memory access while maintaining model quality. As listed in Table 4, our method can achieve imperceptible BLEU score degradation while speedup inference by 1.75x for the first two LSTM layers and 1.70x for the last two LSTM layers. When compromising more translation quality, i.e., decreasing the BLEU score by 2.4, our method can achieve more than 2x speedup. Dimension reduction is an integral part of our dual-module inference method to reduce the number of parameters and memory footprint. Here, we study the impact of different levels of dimension reduction on the model quality and performance. We conduct experiments on language modeling using single-layer LSTM of 1500 hidden units. We quantize the little module to INT8 and reduce the hidden dimension from 1500 to three different levels, which are calculated by Sparse Random Projection. We fix the insensitive ratio to be 50% across this set of experiments. As we can see in Table 5, the higher dimension of the little module, the better approximation the little module can perform. For instance, when we reduce hidden size to 966 and quantize to INT8, the dual-module inference can achieve slightly better quality -PPL of 80.40 -and 1.37x speedup. More aggressive dimension reduction can further have more speedup at the cost of more quality degradation: hidden dimension reduced to 417 and 266 can have 1.67x and 1.71x speedup but increase PPL by 0.72 and 2.87, respectively. We further show the overhead of performing the computation of the little module. As listed in the last three columns in Table 5, we measure the execution time of performing dimension reduction on inputs by Sparse Random Projection, computation of the little module, and computation of the big module; the execution time is normalized to the baseline case, i.e., the execution time of standard LSTM, to highlight the percentage of overheads. When the hidden dimension is reduced to 966, the overhead of the little module accounts 22% while the execution time of the big module is cut off by half 3. In our experiments, we choose = 0.5 as the default parameter in sparse random projection as it demonstrated good quality and speedup trade-off by our study. When further reducing the hidden dimension to 266, there is only a slight improvement on speedup compared with the hidden size of 417 in the little module, where the overhead of the little module is already small enough, but the quality dropped significantly. Quantizing the weights of the little module is another integral part of keeping memory footprint small. We show different quantization levels the impact on model quality and parameter size. After training the little module, we can quantize its weights to lower precision to reduce the memory accessing on top of dimension reduction. As we can see in Table 6, more aggressive quantization leads to smaller parameter size that can reduce the overhead of computing the little module; on the other hand, the approximation of the little module is compromised by quantization. We can quantize the little module up to INT4 without significant quality degradation. Using lower precision would degrade the quality while decreasing the parameter size. For performance evaluation, we choose INT8 as the quantization level since we leverage off-the-shelf INT8 GEMM kernel in MKL. We expect more speedup once the little module overhead can be further reduced by leveraging INT4 compute kernels. As we aim at the memory-bound problem of RNN-based inference applications, we limit the discussion on related work to RNN inference acceleration. Although we only evaluate our dual-module inference method on standard LSTMs/GRUs, we believe our method can be applied to many newly released sequence modeling networks (; as we leverage the commonly observed error-resilience of non-linear activation functions. Compressing DNN models via data quantization, weight sparsity, and knowledge distillation is promising to deliver efficient deployment for inference. propose a quantization method for RNNs where both weights and activations are quantized to binary or ternary. propose a hybrid ternary quantization method based on the different distributions of weights and activations. Weight pruning, i.e., inducing weight sparsity, has been proposed to reduce the parameter size of a pre-trained model (b; a). While fine-grained pruning at element-wise could reduce the number of parameters (; ;), indexing nonzero weights causes extra memory cost and would offset the benefits of reducing parameter size; it is hard to gain practical acceleration on general-purpose hardware or need hardware specialization . Although structural pruning and knowledge distillation could achieve speedup, the applicability on more complicated tasks such NMT using large-scale dataset is unstudied; besides, those methods require extensive retraining via regularization that would increase the training cost and hard to find a solution. Model compression would inevitably compromise the compressive power of RNNs. Our method, by no means, is supposed to replace model compression but provides an orthogonal approach to accelerate RNN inference. Using the analogy of knowledge distillation, we do not simply deploy a student network learned from the teacher network. Instead, we let the teacher network, applied with model compression or not, help with the student -the little module learned from the base moduleand collaboratively perform inference with reduced memory access and computation. Instead of model compression, many work propose to skip computations dynamically based on certain criterion. propose dynamic execution with layer-wise early exit. leverage a special feature of LSTM that using threshold-based pruning on output gates and generate a mask, and then using the mask to skip computation as well as data access of masked-out neurons of the other three gates. utilize temporal input sparsity but need to enforce input similarity with threshold clipping. selectively skip updating the hidden states for some inputs. However, these work either depend on special cell structure or rely on the temporal similarity of inputs which is not evaluated on NLP tasks such as NMT. We are the first that propose a general and principled method to reduce memory access and computation of Gated RNNs, including both LSTMs and GRUs. In this paper, we describe a big-little dual-module inference method to mitigate the memory-bound problem in serving RNN-based models under latency-sensitive scenarios. We leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation to compute for the sensitive region. With overall memory access reduced by near half, our method can achieve 1.54x to 1.75x wall-clock time speedup without significant degradation on model quality. We compare our proposed dual-module inference approach with the automated gradual pruning method , which is a popular pruning method with open implementation 4. Firstly, compared with weight pruning, our method achieves better quality with practical speedup -1.54x to 1.75x reduction on wall-clock time -on commodity CPUs while element-wise weight pruning requires specialized hardware to gain real speedup of computation given irregular sparsity. Moreover, our dual-module inference method can be further applied on top of pruned models to reduce execution time by reducing memory access.

represent the short summary for finding the corresponding scientific article (long).

We accelerate RNN inference by dynamically reducing redundant memory access using a mixture of accurate and approximate modules.

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the scientific article to find a short TLDR

Question: Fine-grained Entity Recognition (FgER) is the task of detecting and classifying entity mentions to a large set of types spanning diverse domains such as biomedical, finance and sports. We observe that when the type set spans several domains, detection of entity mention becomes a limitation for supervised learning models. The primary reason being lack of dataset where entity boundaries are properly annotated while covering a large spectrum of entity types. Our work directly addresses this issue. We propose Heuristics Allied with Distant Supervision (HAnDS) framework to automatically construct a quality dataset suitable for the FgER task. HAnDS framework exploits the high interlink among Wikipedia and Freebase in a pipelined manner, reducing annotation errors introduced by naively using distant supervision approach. Using HAnDS framework, we create two datasets, one suitable for building FgER systems recognizing up to 118 entity types based on the FIGER type hierarchy and another for up to 1115 entity types based on the TypeNet hierarchy. Our extensive empirical experimentation warrants the quality of the generated datasets. Along with this, we also provide a manually annotated dataset for benchmarking FgER systems. In the literature, the problem of recognizing a handful of coarse-grained types such as person, location and organization has been extensively studied BID18 , ]. We term this as Coarse-grained Entity Recognition (CgER) task. For CgER, there exist several datasets, including manually annotated datasets such as CoNLL BID28 ] and automatically generated datasets such as WP2 BID21. Manually constructing a dataset for FgER task is an expensive and time-consuming process as an entity mention could be assigned multiple types from a set of thousands of types. In recent years, one of the subproblems of FgER, the Fine Entity Categorization or Typing (Fine-ET) problem has received lots of attention particularly in expanding its type coverage from a handful of coarse-grained types to thousands of fine-grained types BID17 BID6. The primary driver for this rapid expansion is exploitation of cheap but fairly accurate annotations from Wikipedia and Freebase BID4 via the distant supervision process BID7. The Fine-ET problem assumes that the entity boundaries are provided by an oracle. We observe that the detection of entity mentions at the granularity of Fine-ET is a bottleneck. The existing FgER systems, such as FIGER BID12, follow a two-step approach in which the first step is to detect entity mentions and the second step is to categorize detected entity mentions. For the entity detection, it is assumed that all the fine-categories are subtypes of the following four categories: person, location, organization and miscellaneous. Thus, a model trained on the CoNLL dataset BID28 ] which is annotated with these types can be used for entity detection. Our analysis indicates that in the context of FgER, this assumption is not a valid assumption. As a face value, the miscellaneous type should ideally cover all entity types other than person, location, and organization. However, it only covers 68% of the remaining types of the FIGER hierarchy and 42% of the TypeNet hierarchy. Thus, the models trained using CoNLL data are highly likely to miss a significant portion of entity mentions relevant to automatic knowledge bases construction applications. Our work bridges this gap between entity detection and Fine-ET. We propose to automatically construct a quality dataset suitable for the FgER, i.e, both Fine-ED and Fine-ET using the proposed HAnDS framework. HAnDS is a three-stage pipelined framework wherein each stage different heuristics are used to combat the errors introduced via naively using distant supervision paradigm, including but not limited to the presence of large false negatives. The heuristics are data-driven and use information provided by hyperlinks, alternate names of entities, and orthographic and morphological features of words. Using the HAnDS framework and the two popular type hierarchies available for Fine-ET, the FIGER type hierarchy BID12 and TypeNet BID17, we automatically generated two corpora suitable for the FgER task. The first corpus contains around 38 million entity mentions annotated with 118 entity types. The second corpus contains around 46 million entity mentions annotated with 1115 entity types. Our extensive intrinsic and extrinsic evaluation of the generated datasets warrants its quality. As compared with existing automatically generated datasets, supervised learning models trained on our induced training datasets perform significantly better (approx 20 point improvement on micro-F1 score). Along with the automatically generated dataset, we provide a manually annotated corpora of around thousand sentences annotated with 117 entity types for benchmarking of FgER models. Our contributions are highlighted as follows:• We analyzed that existing practice of using models trained on CoNLL dataset has poor recall for entity detection in the Fine-ET setting, where the type set spans several diverse domains. (Section 3)• We propose HAnDS framework, a heuristics allied with the distant supervision approach to automatically construct datasets suitable for FgER problem, i.e., both fine entity detection and fine entity typing. (Section 4)• We establish the state-of-the-art baselines on our new manually annotated corpus, which covers 2.7 times more finer-entity types than the FIGER gold corpus, the current de facto FgER evaluation corpus. (Section 5)The rest of the paper is organized as follows. We describe the related work in Section 2, followed by a case study on entity detection problem in the Fine-ET setting, in Section 3. Section 4 describes our proposed HAnDS framework, followed by empirical evaluation of the datasets in Section 5. In Section 6 we conclude our work. We majorly divide the related work into two parts. First, we describe work related to the automatic dataset construction in the context of the entity recognition task followed by related work on noise reduction techniques in the context of automatic dataset construction task. In the context of FgER task, BID12 proposed to use distant supervision paradigm BID2 ] to automatically generate a dataset for the Fine-ET problem, which is a sub-problem of FgER. We term this as a Naive Distant Supervision (NDS) approach. In NDS, the linkage between Wikipedia and Freebase is exploited. If there is a hyperlink in a Wikipedia sentence, and that hyperlink is assigned to an entity present in Freebase, then the hyperlinked text is an entity mention whose types are obtained from Freebase. However, this process can only generate positive annotations, i.e., if an entity mention is not hyperlinked, no types will be assigned to that entity mention. The positive only annotations are suitable for Fine-ET problem but it is not suitable for learning entity detection models as there are large number of false negatives (Section 3). This dataset is publicly available as FIGER dataset, along with a manually annotated evaluation corpra. The NDS approach is also used to generate datasets for some variants of the Fine-ET problem such as the Corpus level Fine-Entity typing BID33 and FineEntity typing utilizing knowledge base embeddings BID31. Much recently, BID6 ] generated an entity typing dataset with a very large type set of size 10k using head words as a source of distant supervision as well as using crowdsourcing. In the context of CgER task, BID19 BID20 BID21 proposed an approach to create a training dataset for CgER task using a combination of bootstrapping process and heuristics. The bootstrapping was used to classify a Wikipedia article into five categories, namely PER, LOC, ORG, MISC and NON-ENTITY. The bootstrapping requires initial manually annotated seed examples for each type, which limits its scalability to thousands of types. The heuristics were used to infer additional links in un-linked text, however the proposed heuristics limit the scope of entity and non-entity mentions. For example, one of the heuristics used mostly restricts entity mentions to have at least one character capitalized. This assumption is not true in the context for FgER where entity mentions are from several diverse domains including biomedical domain. There are other notable work which combines NDS with heuristics for generating entity recognition training dataset, such as BID1 and BID9. However, their scope is limited to the application of CgER. Our work revisits the idea of automatic corpus construction in the context of FgER. In HAnDS framework, our main contribution is to design data-driven heuristics which are generic enough to work for over thousands of diverse entity types while maintaining a good annotation quality. An automatic dataset construction process involving heuristics and distant supervision will inevitably introduce noise and its characteristics depend on the dataset construction task. In the context of the Fine-ET task BID24 BID10, the dominant noise in false positives. Whereas, for the relation extraction task both false negatives and false positives noise is present BID25 BID22. In this section, we systematically analyzed existing entity detection systems in the setting of Fine-Entity Typing. Our aim is to answer the following question: How good are entity detection systems when it comes to detecting entity mentions belonging to a large set of diverse types? We performed two analysis. The first analysis is about the type coverage of entity detection systems and the second analysis is about actual performance of entity detection systems on two manually annotated FgER datasets.3.1 Is the Fine-ET type set an expansion of the extensively researched coarse-grained types?For this analysis we manually inspected the most commonly used CgER dataset, CoNLL 2003. We analyzed how many entity types in the two popular Fine-ET hierarchies, FIGER and TypeNet are actual descendent of the four coarse-types present in the CoNLL dataset, namely person, location, organization and miscellaneous. The are available in FIG0. We can observe that in the FIGER typeset, 14% of types are not a descendants of the CoNLL types. This share increases in TypeNet where 25% of types are not descendants Table 1: Performance of entity detection models trained on existing datasets evaluated on the FIGER and 1k-WFB-g datasets.of CoNLL types. These types are from various diverse domain, including bio-medical, legal processes and entertainment and it is important in the aspect of the knowledge base construction applications to detect entity mentions of these types. These differences can be attributed to the fact that since 2003, the entity recognition problem has evolved a lot both in going towards finer-categorization as well as capturing entities from diverse domains. For this analysis we evaluate two publicly available state-of-the-art entity detection systems, the Stanford CoreNLP BID15 and the NER Tagger system proposed in BID11. Along with these, we also train a LSTM-CNN-CRF based sequence labeling model proposed in BID13 on the FIGER dataset. The learning models were evaluated on a manually annotated FIGER corpus and 1k-WFB-g corpus, a new in-house developed corpus specifically for FgER model evaluations. The are presented in Table 1.From the , we can observe that a state-of-the-art sequence labeling model, LSTM-CNN-CRF trained on a dataset generated using NDS approach, such as FIGER dataset has lower recall compared with precision. On average the recall is 58% lower than precision. This is primarily because the NDS approach generates positive only annotations and the remaining un-annotated tokens contains large number of entity mentions. Thus the ing dataset has large false negatives. On the other hand, learning models trained on CoNLL dataset (CoreNLP and NER Tagger), have a much more balanced performance in precision and recall. This is because, being a manually annotated dataset, it is less likely that any entity mention (according to the annotation guidelines) will remain un-annotated. However, the recall is much lower (16% lower) on the 1k-WFB-g corpus as on the FIGER corpus. This is because, when designing 1k-WFB-g we insured that it has sufficient examples covering 117 entity types. Whereas, the FIGER evaluation corpus has only has 42 types of entity mentions and 80% of mentions are from person, location and organization coarse types. These also highlight the coverage issue, mentioned in section 3.1. When the evaluation set is balanced covering a large spectrum of entity types, the performance of models trained on the CoNLL dataset goes down because of presence out-of-scope entity types. An ideal entity detection system should be able to work on the traditional as well as other entities relevant to FgER problem, i.e., good performance across all types. A statistical comparison of FIGER and 1k-WFB-g corpus is provided in Table 2.The use of CoreNLP or learning models trained on CoNLL dataset is a standard practice to detect entity mentions in existing FgER research BID12. Our analysis conveys that this practice has its limitation in terms of detecting entities which are out of the scope of the CoNLL dataset. In the next section, we will describe our approach of automatically creating a training dataset for the FgER task. The same learning models, when trained on our generated training datasets will have a better and a balanced precision and recall. The objective of the HAnDS framework is to automatically create a corpus of sentences where every entity mention is correctly detected and is being characterized into one or more entity types. The scope of entities, i.e., what types of entities should be annotated is decided by a type hierarchy, which is one of the inputs of the framework. FIG2 gives an overview of the HAnDS framework. The framework requires three inputs, a linked text corpus, a knowledge base and a type hierarchy. Linked text corpus: A linked text corpus is a collection of documents where sporadically important concepts are hyperlinked to another document. For example, Wikipedia is a large-scale multi-lingual linked text corpus. The framework considers the span of hyperlinked text (or anchor text) as potential candidates for entity mentions. Knowledge base: A knowledge base (KB) captures concepts, their properties, and interconcept properties. Freebase, WikiData BID29 and UMLS BID3 are examples of popular knowledge bases. A KB usually has a type property where multiple fine-grained semantic types/labels are assigned to each concept. Type hierarchy: A type hierarchy (T) is a hierarchical organization of various entity types. For example, an entity type city is a descendant of type geopolitical entity. There have been various hierarchical organization schemes of fine-grained entity types proposed in literature, which includes, a 200 type scheme proposed in BID26, a 113 type scheme proposed in BID12, a 87 type scheme proposed in BID10 and a 1081 type scheme proposed in BID17. However, in our work, we use two such hierarchies, FIGER 2 and TypeNet. FIGER being the most extensively used hierarchy and TypeNet being the latest and largest entity type hierarchy. Automatic corpora creation using distant supervised methods inevitably will contain errors. For example, in the context of FgER, the errors could be at annotating entity boundaries, i.e, entity detection errors, or assigning an incorrect type, i.e., entity linking errors or both. The three-step process in our proposed HAnDS framework tries to reduce these errors. The objective of this stage is to reduce false positives entity mentions, where an incorrect anchor text is detected as an entity mention. To do so, we first categorize all hyperlinks of the document being processed as entity links and non-entity links. Further, every link is assigned a tag of being a referential link or not. Entity links: These are a subset of links whose anchor text represents candidate entity mentions. If the labels obtained by a KB for a link, belongs to T, we categorize that link as an entity link. Here, the T decides the scope of entities in the generated dataset. For example, if T is the FIGER type hierarchy, then the hyperlink photovoltaic cell is not an entity link as its labels obtained by Freebase is not present in T. However, if T is the TypeNet hierarchy, then photovoltaic cell is an entity link of type invention. Non-entity links: These are a subset of links whose anchor text does not represent an entity mention. Since knowledge bases are incomplete, if a link is not categorized as an entity link it does not mean that the link will not represent an entity. We exploit corpus level context to categorize a link as a non-entity link using the following criteria: across complete corpus, the link should be mentioned at least 50 times (support threshold) and at least 50% of times (confidence threshold) with a lowercase anchor text. The intuition of this criteria is that we want to be certain that a link actually represents a non-entity. For example, this heuristic categorizes RBI as a non-entity link as there is no label present for this link in Freebase. Here RBI refers to the term "run batted in", frequently used in the context of baseball and softball. Unlike, BID19 which discards non entity mentions to have capitalized word, our data-driven heuristics does not put any hard constraints. Referential links: A link is said to be referential if its anchor text has a direct caseinsensitive match with the list of allowed candidate names for the linked concept. A KB can provide such list. For example, for an entity Bill Gates, the candidate names provided by Freebase includes Gates and William Henry Gates. However, in Wikipedia, there exists hyperlinks such as Bill and Melinda Gates linking to Bill Gates page, which is erroneous as the hyperlinked text is not the correct referent of the entity Bill Gates. After categorization of links, except for referential entity links, we unlink all other links. Unlinking non-referential links such as Bill and Melinda Gates reduce entity detection errors by eliminating false positive entity mentions. The unlinked text span or a part of it can be referential mention for some other entities, as in the above example Bill and Melinda Gates. FIG2 also illustrates this process where Lahti, Finland get unlinked after this stage. The next stage tries to re-link the unlinked tokens correctly. The objective of this stage is to reduce false negative entity mentions, where an entity mention is not annotated. This is done by linking the correct referential name of the entity mention to the correct node in KB.To reduce entity linking errors, we use the document level context by restricting the candidate links (entities or non-entities) to the outgoing links of the current document being processed. For example, in FIG2, while processing an article about an FinnishAmerican luger Tristan Jeskanen, it is unlikely to observe mention of a 1903 German novel having the same name, i.e., Tristan. To reduce false negative entity mentions, we construct two trie trees capturing the outgoing links and their candidates referential names for each document. The first trie contains all links and the second trie only contains links of entities which are predominantly expressed in lowercase phrases 3 (e.g. names of diseases). For each non-linked uppercase character, we match the longest matching prefix string within the first trie and assign the matching link. In the remaining non-linked phrases, we match the longest matching prefix string within the second trie and assign the matching link. Linking the candidate entities in unlinked phrases reduces entity detection error, by eliminating false negative entity mentions. Unlike BID19, the two step string matching process ensures the possibility of a lowercase phrase being an entity mention (e.g. lactic acid, apple juice, bronchoconstriction, etc.) and a word with a first uppercase character being a non-entity (e.g. Jazz, RBI, 4 etc.). FIG2 shows an example of the input and output of this stage. In this stage, the phrases Tristan, Lahti, Finland and Jeskanen gets linked. The objective of this stage is to further reduce entity detection errors. This stage is motivated by the incomplete nature of practical knowledge bases. KBs do not capture all entities present in a linked text corpus and do not provide all the referential names for an entity mention. Thus, after stage-II there will be still a possibility of having both types of entity detection errors, false positives, and false negatives. To reduce such errors in the induced corpus, we select sentences where it is most likely that all entity mention are annotated correctly. The ant corpora of selected sentences Table 2: Statistics of the different datasets generated or used in this work.will be our final dataset. To select these sentences, we exploit sentence-level context by using POS tags and list of the frequent sentence starting words. We only select sentences where all unlinked tokens are most likely to be a non-entity mention. If an unlinked token has a capitalized characters, then it likely to be an entity mention. We do not select such sentences, except in the following cases. In the first case, the token is a sentence starter, and is either in a list of frequent sentence starter word 5 or its POS tag is among the list of permissible tags 6. In the second case, the token is an adjective, or belongs to occupational titles or is a name of day or month. FIG2 shows an example of the input and output of this stage. Here only the first sentence of the document is selected because in the other sentence the name Sami is not linked. The sentence selection stage ensures that the selected sentences have high-quality annotations. We observe that only around 40% of sentences are selected by stage III in our experimental setup. 7 Our extrinsic analysis in Section 5.2 shows that this stage helps models to have a significantly better recall. In the next section, we describe the dataset generated using the HAnDS framework along with its evaluations. Using the HAnDS framework we generated two datasets as described below: WikiFbF: A dataset generated using Wikipedia, Freebase and the FIGER hierarchy as an input for the HAnDS framework. This dataset contains around 38 million entity mentions annotated with 118 different types. WikiFbT: A dataset generated using Wikipedia, Freebase and the TypeNet hierarchy as an input for the HAnDS framework. This dataset contains around 46 million entity mentions annotated with 1115 different types. In our experiments, we use the September 2016 Wikipedia dump. Table 2 lists various statistics of these datasets. In the next subsections, we estimate the quality of the generated datasets, both intrinsically and extrinsically. Our intrinsic evaluation is focused on 5. 150 most frequent words were used in the list. 6. POS tags such as DT, IN, PRP, CC, WDT etc. that are least likely to be candidate for entity mention.7. An analysis of several characteristics of the discarded and retained sentences in available in the supplementary material at: https://github.com/abhipec/HAnDS. Table 3: Quantitative analysis of dataset generated using the HAnDS framework with the NDS approach of dataset generation. Here H denotes a set of entity mentions in Table 3a and set of entities in Table 3b generated by the HAnDS framework, and N denotes a set of entity mentions in Table 3a and set of entities in Table 3b generated by the NDS approach.quantitative analysis, and the extrinsic evaluation is used as a proxy to estimate precision and recall of annotations. In intrinsic evaluation, we perform a quantitative analysis of the annotations generated by the HAnDS framework with the NDS approach. The of this analysis is presented in Table 3. We can observe that on the same sentences, HAnDS framework is able to generate about 1.9 times more entity mention annotations and about 1.6 times more entities for the WikiFbT corpus compared with the NDS approach. Similarly, there are around 1.8 times more entity mentions and about 1.6 time more entities in the WikiFbF corpus. In Section 5.2.4, we will observe that despite around 1.6 to 1.9 times more new annotations, these annotations have a very high linking precision. Also, there is a large overlap among annotations generated using HAnDS framework and NDS approach. Around above 95% of entity mentions (and entities) annotations generated using the NDS approach are present in the HAnDS framework induced corpora. This indicated that the existing links present in Wikipedia are of high quality. The remaining 5% links were removed by the HAnDS framework as false positive entity mentions. In extrinsic evaluation, we evaluate the performance of learning models when trained on datasets generated using the HAnDS framework. Due to resource constraints, we perform this evaluation only for the WikiFbF dataset and its variants. Following BID12 we divided the FgER task into two subtasks: Fine-ED, a sequence labeling problem and Fine-ET, a multi-label classification problem. We use the existing state-of-the-art models for the respective sub-tasks. The FgER model is a simple pipeline combination of a Fine-ED model followed by a Fine-ET model. Fine-ED model: For Fine-ED task we use a state-of-the-art sequence labeling based LSTM-CNN-CRF model as proposed in BID13.Fine-ET model: For Fine-ET task we use a state-of-the-art LSTM based model as proposed in BID0. Please refer to the respective papers for model details. 8 The values of various hyperparameters used in the models along with the training procedure is mentioned in the supplementary material available at: https://github.com/abhipec/HAnDS. The two learning models are trained on the following datasets: Wiki-FbF: Dataset created by the HAnDS framework. Wiki-FbF-w/o-III: Dataset created by the HAnDS framework without using stage III of the pipeline. Wiki-NDS: Dataset created using the naive distant supervision approach with the same Wikipedia version used in our work.. FIGER: Dataset created using the NDS approach but shared by BID12.Except for the FIGER dataset, for other datasets, we randomly sampled two million sentences for model training due to computational constraints. However, during model training as described in the supplementary material, we ensured that every model irrespective of the dataset, is trained for approximately same number of examples to reduce any bias introduced due to difference in the number of entity mentions present in each dataset. All extrinsic evaluation experiments, subsequently reported in this section are performed on these randomly sampled datasets. Also, the same dataset is used to train Fine-ED and Fine-ET learning model. This setting is different from BID12 where entity detection model is trained on the CoNLL dataset. Hence, the reported in their work is not directly comparable. We evaluated the learning models on the following two datasets: FIGER: This is a manually annotated evaluation corpus which has been created by BID12. This contains 563 entity mentions and overall 43 different entity types. The type distribution in this corpus is skewed as only 11 entity types are mentioned more than 10 times. 1k-WFB-g: This is a new manually annotated evaluation corpus developed specifically to cover large type set. This contains 2420 entity mentions and overall 117 different entity types. In this corpus 84 entity types are mentioned more than 10 types. The sentences for this dataset construction were sampled from Wikipedia text. The statistics of these datasets is available in Table 2. For the Fine-ED task, we evaluated model's performance using the precision, recall and F1 metrics as computed by the standard conll evaluation script 9. For the Fine-ET and 8. Please note that there are several other models with competitive or better performance such as BID5 BID11 BID13 for sequence labeling problem and BID23 BID27 BID0 BID31 BID32 for multi-label classification problem. Our criteria for model selection was simple; easy to use publicly available efficient implementation. Table 4: Performance of the entity detection models on the FIGER and 1k-WFB-g datasets.the FgER task, we use the strict, loose-macro-average and loose-micro-average evaluation metrics described in BID12. The of the entity detection models on the two evaluation datasets are presented in Table 4. From these we perform two analysis. First, the effect of training datasets on model's performance and second, the performance comparison among the two manually annotated datasets. In the first analysis, we observe that the LSTM-CNN-CRF model when trained on WikiFbF dataset has the highest F1 score on both the evaluation corpus. Moreover, the average difference in precision and recall for this model is the lowest, which indicates a balanced performance across both evaluation corpus. When compared with the models trained on the NDS generated datasets (Wiki-NDS and FIGER), we observe that these models have best precision across both corpus, however, lowest recall. The indicates that large number of false negatives entity mentions are present in the NDS induced datasets. In the case of model trained on the dataset Wiki-FbF-w/o-III dataset the performance is in between the performance of model trained on Wiki-NDS and Wiki-FbF datasets. However, they have a significantly lower recall on average around 28% lower than model trained on Wiki-FbF. This highlights the role of stage-III, by selecting only quality annotated sentence, erroneous annotations are removed, ing in learning models trained on WikiFbF to have a better and a balanced performance. In the second analysis, we observe that models trained on datasets generated using Wikipedia as sentence source, performs better on the 1k-WFB-g evaluation corpus as compared to the FIGER evaluation corpus. These datasets are FIGER training corpus, WikiFbF, Wiki-NDS and Wiki-FbF-w/o-III. The primarily reason for better performance is that the sentences constituting the 1k-WFB-g dataset were sampled from Wikipedia. 10 Thus, this evaluation is a same domain evaluation. On the other hand, FIGER evaluation corpus is based on sentences sampled from news and specialized magazines (photography and veterinary domains). It has been observed in the literature that in a cross domain evaluation setting, learning model performance is reduced compared to the same domain evaluation BID20. Moreover, this also conveys that to some extent learning model trained on the large Wikipedia text corpus is also able to generalize on evaluation dataset consisting of sentences from news and specialized magazines. Our analysis in this section as well as in Section 3.1 indicates that although the type coverage of FIGER evaluation corpus is low (43 types), it helps to better measure model's generalizability in a cross-domain evaluation. Whereas, 1k-WFB-g helps to measure performance across a large spectrum of entity types (117 types). Learning models trained on Wiki-FbF perform best on both of the evaluation corpora. This warrants the usability of the generated corpus as well as the framework used to generate the corpus. We observe that for the Fine-ET task, there is not a significant difference between the performance of learning models trained on the Wiki-NDS dataset and models trained on the Wiki-FbF dataset. The later model performs approx 1% better in the micro-F1 metric computed on the 1k-WFB-g corpus. This indicates that in the HAnDS framework stage-II, where false negative entity mentions were reduced by relinking them to Freebase, has a very high linking precision similar to NDS, which is estimated to be about 97-98% BID30.The for the complete FgER system, i.e., Fine-ED followed by Fine-ET are available in TAB5. These supports our claim in Section 3.1, that the current bottleneck for the FgER task, is Fine-ED, specifically lack of resource with quality entity boundary annotations while covering large spectrum of entity types. Our work directly addressed this issue. In the FgER task performance measure, learning model trained on WikiFbF has an average absolute performance improvement of at least 18% on all of the there evaluation metrics. In this work, we initiate a push towards moving from CgER systems to FgER systems, i.e., from recognizing entities from a handful of types to thousands of types. We propose the HAnDS framework to automatically construct quality training dataset for different variants of FgER tasks. The two datasets constructed in our work along with the evaluation resource are currently the largest available training and testing dataset for the entity recognition problem. They are backed with empirical experimentation to warrants the quality of the constructed corpora. The datasets generated in our work opens up two new research directions related to the entity recognition problem. The first direction is towards an exploration of sequence labeling approaches in the setting of FgER, where each entity mention can have more than one type. The existing state-of-the-art sequence labeling models for the CgER task, can not be directly applied in the FgER setting due to state space explosion in the multi-label setting. The second direction is towards noise robust sequence labeling models, where some of the entity boundaries are incorrect. For example, in our induced datasets, there are still entity detection errors, which are inevitable in any heuristic approach. There has been some work explored in BID8 assuming that it is a priori known which tokens have noise. This information is not available in our generated datasets. Additionally, the generated datasets are much richer in entity types compared to any existing entity recognition datasets. For example, the generated dataset contains entities from several domains such as biomedical, finance, sports, products and entertainment. In several downstream applications where NER is used on a text writing style different from Wikipedia, the generated dataset is a good candidate as a source dataset for transfer learning to improve domain-specific performance.

Represent the short summary for finding the corresponding scientific article (long)

We initiate a push towards building ER systems to recognize thousands of types by providing a method to automatically construct suitable datasets based on the type hierarchy.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the scientific article to find a short TLDR

Question: Implementing correct method invocation is an important task for software developers. However, this is challenging work, since the structure of method invocation can be complicated. In this paper, we propose InvocMap, a code completion tool allows developers to obtain an implementation of multiple method invocations from a list of method names inside code context. InvocMap is able to predict the nested method invocations which their names didn’t appear in the list of input method names given by developers. To achieve this, we analyze the Method Invocations by four levels of abstraction. We build a Machine Translation engine to learn the mapping from the first level to the third level of abstraction of multiple method invocations, which only requires developers to manually add local variables from generated expression to get the final code. We evaluate our proposed approach on six popular libraries: JDK, Android, GWT, Joda-Time, Hibernate, and Xstream. With the training corpus of 2.86 million method invocations extracted from 1000 Java Github projects and the testing corpus extracted from 120 online forums code snippets, InvocMap achieves the accuracy rate up to 84 in F1- score depending on how much information of context provided along with method names, that shows its potential for auto code completion. Writing code is a challenge for non-experienced software developers. To write the code that implements a specific task in a programming language, developers need to remember the syntax of that language and be familiar with how to implement method invocations. While the syntax of the language is easier to learn since it contains a permanent set of words in the vocabulary, implementing Method Invocations (MI)s is more challenging due to the following reasons. First of all, developers need to remember the structure and the combination of invocations depending on their purpose. Secondly, the implementation of method invocation is also depending on the surrounding context of the code. Thus, the code developed by non-experience developers may be in the risks of being semantic error. To help developers with interacting and analyzing by a given Java source code snippet, Java Development Tool (JDT) library defines a list of Abstract Syntax Tree (AST) Node types . With the list of these AST Node types, JDT is able to interact with the structure of each elements inside the source code. MI, which is defined as sub-type of Expression, is one of the fundamental AST Nodes that developers need to implement. MI has been used to make Application Programming Interface (API) calls from other libraries or from other methods inside a Java project. The structure of a syntactically correct MI contains method name, receiver and the list of arguments which could be empty. Since receiver and arguments are types of expression , the structure of an MI could be complicated as a deep AST tree. The reason for this issue is that expression can be composed by different types of AST Node including MI. An example of a complicated MI is shown in Listing 1. Within this Listing, the outside MI contains four nested MI in its implementation. Additionally, there are five positions that requires local variables inside the expression. Type casting to integer is embedded to this MI to provide a semantically correct MI. This MI is used along with other calculated MIs inside the body of method, providing the a specific surrounding context for this MI. Without doubt, the outer method name set is just one word while the respected MI is a deep AST tree. The representation of MI also relies on code context. Consider examples 2A and 2B on Listing 2 and Listing 3. These Listings show the implementation of API android.content.Intent.getBooleanExtra. Although 2 MIs share the same information about context of using the same local variable Intent and the false boolean literal, they are differ in the structure of AST. Since the MI in Listing 2 associates with the action of add or remove an application package from an android device, the MI on Listing 3 associates with actions of network status checking. The difference in contexts brings 2 MIs, which represents in 2 static Field Accesses Intent. EXTRA REPLACING and ConnectivityManager. EXTRA NO CONNECTIVITY. Listing 1: Example in Android (2019a) 1 p u b l i c v o i d s e t O f f s e t s (i n t n e w H o r i z o n t a l O f f s e t, i n t n e w V e r t i c a l O f f s e t) {2 . . . . . . 5 i n v a l i d a t e R e c t f . o f f s e t (− x o f f s e t, −y o f f s e t); 6 i n v a l i d a t e R e c t. s e t (( i n t) Math. f l o o r (i n v a l i d a t e R e c t f . l e f t), (i n t) Math. f l o o r (i n v a l i d a t e R e c t f . t o p), (i n t) Math. c e i l (i n v a l i d a t e R e c t f . r i g h t), (i n t) Math. c e i l (i n v a l i d a t e R e c t f . b o t t o m) ); 7... Listing 2: Example 2A in Android (2019b) 1 p u b l i c v o i d o n R e c e i v e (C o n t e x t c o n t e x t, I n t e n t i n t e n t) {2 . . . 3 i f ( ( I n t e n t . ACTION PACKAGE REMOVED . e q u a l s ( a c t i o n) | | 4 I n t e n t. ACTION PACKAGE 5 ADDED. e q u a l s (a c t i o n) ) 6 &&! i n t e n t. g e t B o o l e a n E x t r a (I n t e n t . EXTRA REPLACING, f a l s e) ) {7 . . . Listing 3: Example 2B in Android (2019c) 1 p u b l i c v o i d o n R e c e i v e (C o n t e x t c o n t e x t, I n t e n t i n t e n t) {2 . . . 3 i f ( a c t i v e N e t w o r k == n u l l) {4 . . . 5} e l s e i f (a c t i v e N e t w o r k . g e t T y p e == n e t w o r k T y p e ) {6 mNetworkUnmetered = f a l s e ; 7 mNetworkConnected = ! i n t e n t . g e t B o o l e a n E x t r a ( C o n n e c t i v i t y M a n a g e r . EXTRA NO CONNECTIVITY, f a l s e); 8... From the examples above, we recognize that implementing an effective method invocation requires strong and experiences of developers. Even two MIs that belong to the same API and share the same context of local variables and literal still have ambiguous in the way of implementation like Listing 2 and Listing 3. These challenges hinders the ability of writing a appropriate MI and as well as developers need to spend time to remember or identify the correct structure of AST in MI for software development. With this work, we want to tackle this problem by providing InvocMap, a code completion tool for helping developers to achieve the implementation of method invocation efficiently. InvocMap accepts input as a sequence of method names inside the code environment of a method declaration, then produce the output as the list of ASTs as translation for each input method names. The generated ASTs will only require developers to input information about local variables and literals in order to obtain the complete code. For instance, in Listing 2, developer can write the list of method names including the name getBooleanExtra. The output for the suggestion will be #.getBooleanExtra(Intent. EXTRA REPLACING,#), which can be completed manually by a variable of type android.content.Intent in the first "#" and a boolean literal in the second "#". Statistical Machine Translation (SMT) is a well-known approach in Natural Language Processing (NLP) for translating between languages . For taking advantage from SMT, we propose a direction of code completion for Method Invocation by a Statistical approach, which learn the translation from the abstract information of MIs to the their detail information, which are represented by AST with complicate structure. First and foremost, we analyze the information inside a typical MI. We divide the MI by four levels of abstraction. We also define information of context for each MI which can help to predict the AST structure. Next, we build an SMT engine specified for our work to infer from the very abstract layer of MI, means Method Name, to the third level of MI, which is an AST tree that requires to be fulfill by local variables and literals. In order to evaluate our approach, we do experiments to check the accuracy of our code completion technique in two data sets collected from Github and from online forums. Resources of this paper can be found in . This research has following contributions: 2. Designing rules for extracting code tokens for representing abstract level and details level for various types of AST nodes. 3. Proposing an algorithm for visiting a method invocation inside the code environment to abstract and encode their structure in AST as an object for statistical learning. 4. Building a SMT system for learning from the context of code environment, including MIs from large scale Github high quality projects. This SMT system is able to predict the sequences of AST structure given sequences of method name and context. We summarize the engines inside InvocMap on Figure 1. From the perspective of developers, InvocMap provides a plugin inside with Java code editor to allow them to write a single or multiple method names inside the code environment. Starting with this input, InvocMap translates each method names to respective ASTs. These ASTs reflect the complex structure of method invocations which might be inconvenient for developers to remember. They are abstracted at level 3 in our definition. That means they only require developers to add local variables, local methods or literals to obtain the final code. We will discuss about MI at level 3 of abstraction in the next section. The ability of inferring ASTs for code completion relies on the Statistical Translation module. The training process is done by the Statistical Learning module. This module learns information from the data extracted from large scale Github code corpus . In general, our statistical approach takes advantages of the knowledge of implementing MIs from experienced developers, representing it by a machine learning model to help non-experienced developers in retrieving effective implementation of MIs. Both the source code at developers side and code corpus are analyzed to extract sequences of tokens by the Train AST Visitor and Test AST Visitor modules we developed. Inside these visitors, we handle each AST Node types by functions of module Context Extractor and MI Abstractor, which we discuss in next sections. Definition 1 Level 1 of abstraction of a method invocation is the information about method name of that method invocation. Definition 2 Level 2 of abstraction of a method invocation is the information about type (or signature) of that method invocation. Definition 3 Level 3 of abstraction of a method invocation is the Abstract Syntax Tree of that method invocation with abstracted place holder for local variables, local methods and literal. Definition 4 Level 4 of abstraction of a method invocation is the complete Abstract Syntax Tree of that method invocation. Along with 4 levels of abstraction in MI, we have the definition of local context provided for each MI. An example of 4 levels is shown in Figure 2 (a). In this code snippet, we have level 1 as method name println. The level 2 of abstraction brings us information about type, which is java.io.PrintStream.println.The level 4 is the final source code which is compile-able. The level 3 is the AST that is having places which are local entities are abstracted by their type information. In the implementation, we represent this AST in level 3 by 4 fields: the code with abstracted places for local entities, the list of types of required arguments to add to get level 4, the list of imported APIs and the type of MI. These 4 fields will make an unique identification for the expression, which will serve as a representative token for the AST. Therefore, developers could know which types of local variables to obtain the final code along with the set of imported APIs when they receive an AST at level 3 of abstraction. In our work, we focus on the inference from level 1 to level 3 by translation. We will use information of local context to help developers who already remember what variables should run inside the MI and some words inside the MI to better retrieve the AST of implementation. In Figure 2 (a), we see 2 local entities, including the string literal "index" and the integer variable i. The suggested terms can be "System" and "+" sign. Definition 6 Level 1 of abstraction of other AST Nodes is the information about the Partial Qualified Name (PQN) of type of those nodes. Definition 7 Level 2 of abstraction of other AST Nodes is the information about Fully Qualified Name (FQN) of type of those nodes. In the context of this work, we call other AST Nodes as all kinds of AST except the MI that are defined in. According to definitions of , an example is the API java.io.File. In this API, we have File as PQN while we have java.io.File as FQN. Other AST Nodes tokens. We extract information about other AST Nodes to provide useful context for MIs prediction. In the source language, we extract all tokens of level 1 of abstraction for each AST Node, and extract all tokens in level 2 of that AST Node to put into target language. The implementation of the extraction is the Context Extractor module, which is called inside Train AST Visitor and Test AST Visitor. MI tokens. There are two types of information we want to embed for MI: the mapping between method name and the AST along with the information relate to local context. For the first type of information, the source language will store information about token as level 1 of abstraction of MI, while the target language stores information about level 3 of abstraction of MI. Besides, information about local context will be stored by level 1 of abstraction in the source and level 2 of abstraction in the target language. A sequence of tokens for MI in Figure 2 (a) is shown in Figure 2 } 21 e l s e {22 r e s u l t . s e t I m p o r t e d A P I s . add ( g e t T y p e ( node) ); 27 } 28 } We get information about level 3 of abstraction in MI by proposing an algorithm in Listings 4 and 5. The abstractMethodInvocation function is invoked when the Train AST Visitor or Test AST Visitor visit a MI and return the abstraction in level 3 by an instance of AST Level3 class. This function will use the child class of ASTVisitor called InvocAbstractVisitor defined in Listing 5 (line #12). This visitor will visit each element inside the MI, check and abstract if the element is a local entity. This visitor also stores other information about the code of AST, the list of required types for each local entities and the set of imported APIs. The handling strategy for each types of AST Node inside the MI is implemented in the visitStructure function in Listing 5(#23). After visiting and abstracting of MI to an AST Level3, this object is checked by the first four fields defined in Listing 5(#1-#10) to see if its exist in the dictionary or not. If yes, it will have the id of the existing object in the dictionary. Otherwise, it will generate a new unique id and will be added to the dictionary. The dictionary stores information about abstraction at layer 3 of MIs in the training step. An example of AST Level3 object is shown in Figure 2 (a). To learn the mapping between source and target language, we apply the SMT . SMT was built from two models: the language model and the translation model.. LM is used to predict the next token given a sequence of previous tokens . The more comprehensive corpus of target language we have, the higher quality of prediction the LM achieves. LM had been used widely in Software Engineering (SE) researches (; ;) with potential . The most basic LM in NLP is uni-gram LM, which calculates the probability of each word based on the number of the appearance of that word in the corpus. This LM provides drawbacks that it doesn't take into account the history of how a word was used in a sequence from training data. Here we use the n-gram language model, which proposed by. Assume that we have m tokens in the target language AST 1,..., AST m, the probability provided by LM is shown in the above equation of Equations 1. Translation Model. This model calculates the probability of a phrase from source language that can be translated to a phrase in a target language. If we have a sentence D as the translated of sentence S as tokens in the source language, the selection of D as the best candidate is calculated by the below equation in Equations 1. Since we infer from method names to MIs which are consistent in order, we don't apply the reordering probability in the translation model. Data Preparation. To do the evaluation, we select corpus on six well-known libraries. They are Java Development Kit (JDK), Android, GWT, Joda-Time, Hibernate, and XStream. These libraries were selected to generate the corpus for other research works . To generate corpus, we select 1000 highest stars Java projects from Github , which have most files used APIs from libraries in Table 1a. For each Java project, InvocMap parses each Java source files by an the Train ASTVisitor module on Figure 1. The number of pairs respected to each method body we collect is shown in Table 1a. Training and Testing Configuration. To train the SMT model, we use a high-end computer with core-i7 Intel processor and use 32 GB of memory. We apply our solution using. We allocate Phrasal with phrase length equals to 7. The total training time requires about 6 hours. For testing, we evaluate the ability of translation from a sequence of method names to ASTs in level 3 of abstraction. We simulate 3 configurations sequences of method names regarding to its local context defined in Table 1b. We can see the local context provided for method names is increasing from configurations at level 1 to level 3. At level 1, the input for translation contains only method names with the code context in the source language for translation. It simulates the case that developers write a list of method names inside the code environment. At level 2, information about partial class name of types of local entities is attached along with each method names. This case simulates the case developers remember and write method name and local variables they remember as part of the MI, but they don't remember the structure of AST. At level 3, each method names in the source language will be attached the information about local entities and half of words appeared inside the MI. This case simulates the case that developers remember some words inside the MI along with local entities. Metrics. Information about tokens of method name and MI can be recognized by the annotation #identifier in the source, and the expected can be recognized by prefix "E-Total" of tokens in the target. We use Precision and Recall as 2 metrics for the evaluation. Out of Vocabulary (OOV) is the case that the method name token does not in the corpus (Out of Source -OOS) or the expected AST in level 3 does not appear in the target corpus (Out of Target -OOT). We split the pairs of our parallel corpus for training and testing. We get 10% of the data for testing and the other with training and do ten-fold cross-validation to test the ability of prediction on our full data set. In total, there are 2.86 Million of MIs collected from 1000 projects from. The evaluation for intrinsic data is shown in Table 2. We show that from configuration 1 to configuration 3, the F1 score increases from 73.06% to 84.4%. This seems to be feasible, since the fact that if we provide more local context information along with method names, the ability to predict correctly AST in level 3 for the translation model is better. We see one observation is that the number of Out of Vocabulary expressions are higher in percentage, cause decreasing in recall compare to the research work that applied Machine Translation for inferring Fully Qualified Name from incomplete code . This is reasonable, since our work requires to infer the MI in level 3 of abstraction, which contains detail structure compared to output of , which only infers the type information of MI. We study an example in the Intrinsic Evaluation in Figure 2 (b). This example is a function collected from from our corpus. The testing for intrinsic evaluation simulates the case developers input only println inside the code environment, the output of this case will be the implementation of java.io.PrintWriter.println function. We can see that the surrounding code is useful to infer the correct expression. If we do not have the context information, which means developer input println in an empty method, the translated will return the most popular MI, System.out.println. To do this experiment, we collect the data as code snippets from Online Forums (; ;). A Software Engineer who has 5 years of experience in Java programming was hired to collect code snippets from 120 posts in Online Forums, with 20 posts for each library in Table 1a. The for extrinsic evaluation is shown in Table 2. We see that with level 1, since the case that only method names are provided in the source language, our approach stills predict correctly 68.5% in F1-score. With the configuration levels that developers add more information, the F1-score increases to 84%. For each library, we achieved the highest accuracy on GWT and lowest on Hibernate with input as detail information like configuration 3. This seems reasonable, since Hibernate is a bigger library compared to GWT but it is not as popular as JDK, causes the variety of ASTs for APIs in this library. In this evaluation, we analyze the relation of the expression prediction relates to the number of mapping of each method name from the parallel corpus. We use data collected for the Intrinsic Evaluation with configuration 3. The , which is shown in Table 1c, reveals that from the number of method name that has more than 100 mappings in the parallel corpus are about 72% of the total data. It proves the complexity of kinds of implementation for each method names. The total precision tends to decrease from 96.47 % to 87.68% from low to high number of mappings, means that the prediction is still acceptable although the method names are too ambiguous. Machine Learning has been applied widely in Software Engineering applications. Generating code by machine learning is an interesting but also confront challenges. There is a research by shows that the inference of code from documentation by machine translation achieved very low accuracy on both SMT and Neural Machine Translation (NMT) models learned from practical large scale code corpus. There are two reasons cause to this challenge. First, large scale code corpus contains noise data. Second, the structure of AST Node is complicate for a machine translation system to learn about the syntactically correct of generated code as shown in. propose an approach to achieve the implementation from in natural language description. However, the output of their tool consists only sequence of APIs which is in level 2 of our abstraction for MIs. In our work, we target the inference of MI in level 3 with the ability of complex AST structure of MIs. There are several other inputs to get the complete code in other researches. derive the code in C# language from code in Java language by machine translation. generate the code from natural language descriptions. In these works, they consider the textual description as the full information for the inference. We consider our code generation problem in a different angle, which we take advantage of the surrounding context along with the textual description of method name in our work. propose a graph based code completion tool that suggest the full code snippet when developers are writing an incomplete code. This work focuses on completing the code from a part of the code. We propose an inference from the skeleton of method invocations, which is in form of sequence of method names, to the implementation of method invocations. In this work, we proposed InvocMap, a SMT engine for inferring the ASTs of method invocations from a list of method names and code context. By the evaluation on corpus collected from Github projects and online forums, we demonstrated the potential of our approach for auto code completion. A major advantage of InvocMap is that it is built on the idea of abstracting method invocations by four different levels. We provided an algorithm to achieve AST of method invocations for the method invocations inference. As future works, we will work on extending the SMT model to support inputs from multiple natural language descriptions of multiple method invocations, along with investigation of machine learning techniques for improving the accuracy.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper proposes a theory of classifying Method Invocations by different abstraction levels and conducting a statistical approach for code completion from method name to method invocation.

Represent the TLDR for finding the corresponding scientific article

Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.