Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: In this paper we present a method for algorithmic melody generation using a generative adversarial network without recurrent components. Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies. Here, we use DCGAN architecture with dilated convolutions and towers to capture sequential information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel. Algorithmic music composition is almost as old as computers themselves, dating back to the 1957 "Illiac suite" . Since then, automated music composition evolved with technology, progressing from the first rule-and-randomness based methods to the sophisticated tools made possible by modern-day machine learning (see Fernández & and for detailed surveys on history and state of the art of algorithmic music composition). One of the first machine learning (ML) approaches to music generation was , who used the common notion of entropy as a measurement to build what they termed a multiple viewpoint system. Standard feedforward neural networks have difficulties with sequence based information such as music. Predicting the next note of a piece, when only based on the current note, does not account for long-range context or structure (such as key and musical sections) which help give coherence to compositions. As music is traditionally represented as sequences of notes, recurrent neural networks are a natural tool for music (especially melody) generation, and multiple groups used RNNs fairly successfully for a variety of types of music. used a sequential model for composition in 1989, and used the adapted LSTM structure to successfully generate music that had both short-term musical structure and contained the higher-level context and structure needed. Subsequently, there have been a number of RNN-based melody generators (; ; ; ; ; ; . Other approaches such as MidiNet by , though not RNNs, also leveraged the sequential representation of music. Using an RNN architecture provides a lot of flexibility when generating music, as an RNN has the ability to generate pieces of varying length. However, in some styles of music this is not as desired. This is true of traditional Irish music -and especially their jigs and reels. These pieces have a more rigid format where the varying length can prevent capturing the interplay between the phrases of the piece. Finding jigs and reels to train on was made easy by an excellent database of Irish traditional melodies in ABC notation (a text based format), publicly available at TheSessionKeith. Several RNN-based generators were trained on the melodies from TheSession, most notably Sturm et al. , as well as. It is natural to view music, and in particular melodies, as sequential data. However, to better represent long-term dependencies it can be useful to present music as a two-dimensional form, where related parts and occurrences of long patterns end up aligned. This benefit is especially apparent in forms of music where a piece consists of a well-defined, fixed-length components, such as reels in Irish music. These components are often variations on the same theme, with specific rules on where repeats vs. changes should be introduced. Aligning them allows us to use vertical spatial proximity to capture these dependencies, while still representing the local structure in the sequence by horizontal proximity. In this project, we leverage such two-dimensional representation of melodies for non-sequential melody generation. We focus on melody generation using deep convolutional generative adversarial networks (DCGANs) without recurrent components for fixed-format music such as reels. This approach is intended to capture higher-level structures in the pieces (like sections), and better mimic interplay between smaller parts (musical motifs). More specifically, we use dilations of several semantically meaningful lengths (a bar or a phrase) to further capture the dependencies. Dilated convolutions, introduced by , have been used in a number of applications over the last several years to capture long-range dependencies, notably in WaveNet . However, they are usually combined with some recurrent component even when used for a GAN-based generation such as in or. Not all techniques applicable to images can be used for music, however: pooling isn't effective, as the average of two pitches can create notes which fall outside of the 12-semitone scale (which is the basis the major and minor scale as well as various modes). This is reflected in the architecture of our discriminator, with dilations and towers as the main ingredients. In spite of emigration and a well-developed connection to music influences from its neighbouring cultures -especially Scotland -Irish traditional music has kept many of its elements and itself influenced many forms of music. Its influences can be found in the folk music of Newfoundland and Quebec and in Bluegrass from the Appalachian region of United States. A significant part of the traditional Irish music is instrumental dance music, where a tune following a set format is played on a solo instrument, such as a fiddle, tin whistle, accordion or flute. The core of the piece is usually monophonic, so while harmonies and embellishments can be introduced during performances, they are considered extra and not usually encoded when a piece is transcribed. Generally the melody consists of two distinct 8-bar parts, which in turn consist of two 4-bar phrases each. Usually, phrases represent a pattern which repeats with variations from phrase to phrase, with one (most often third) phrase deviating most from the pattern, and the final phrase returning to it. When performing, each part is repeated twice; sometimes the second repeat of a part is different enough that it is transcribed separately, making the transcription from 16 to 32 bars long. The whole piece is usually performed three (or more) times, especially when accompanying a dance. The two most common types of dances are reels and jigs: the reels have time signature of 4/4 (that is, each bar contains 4 units, where each unit is a quarter note), whereas a jig has a time signature 6/8 (that is, each bar contains 6 units, where each unit is an eighth note). Many Irish tunes are in a major key, with D major being most common. However, a number of tunes use the Dorian and Mixolydian modes (starting with 2nd or 5th note with respect to the relative major scale, respectively). The natural minor key (Aeolian mode) is less common. Overall, it is uncommon for a tune to incorporate notes not in its key and to go outside of a 2-octave range (either of these would make it hard to play on a tin whistle). Traditionally, musicians learned tunes by ear. However, the relative simplicity of the structure of Irish tunes has lent itself well to a text-based representation, culminating in the ABC notation developed by Chris Walshaw in the 1970s. There, the key and default length of a note are specified in a header (together with the title, time signature, etc), and the melody is then encoded by letters: A-G,where C represents middle C, and a-g, which represents the notes an octave higher than A-G. Additionally there are modifiers which raise and lower notes by an octave, as well as charaters which represent sharps and flats. Like in the staff notation, sharps and flats which are implied by the key are not explicitly written into the piece. These letters are also appended with duration modifiers: for example, if ABC header sets the default note length as 1/8, then D represents an eighth note (which is a D), D2 represents a quarter note, and D/2 represents a sixteenth note. For readability, bar separators | and spacing are often included: for example, the sequence "d2dA BAFA|ABdA BAFA|ABde f2ed|Beed egfe" encodes the first phrase of a reel called "Merry Blacksmith", where the key was specified to be D major and the default note length is set to 1/8. Nowadays, ABC notation is widely used, especially for Irish folk music, with additional features for representing ornaments, repetitions, and so on. In particular, this is the default representation for tunes in databases such as TheSession which we used for this project. TheSession dataset contains over 31,000 tunes, which contain 10,000 reels in a major key. However, after removing improperly formatted tunes, tunes with features we did not want to generate (such as triplets), and tunes that had more than 16 bars we were left with 820 reels, which we used as our training data. The samples in ABC notation were then converted into numerical vectors, with 16 numbers per bar (so 256 numbers total), corresponding to 16 possible notes (normalized midi values). This encoding did not preserve information about the duration of each note, only their (normalized) midi value. However, we found that with simple postprocessing (converting a sequence of occurrences of the same note into a note of a longer duration, starting with an occurrence on a beat) we were able to recover the tunes with not much more variation than would be introduced in performances. One of the main ideas behind our encoding was representing the ing vectors of 256 values as a 64x4 image, with midi values corresponding to pixel values. The ing image had a line for each the phrase of the tune, and the vertical alignment of the corresponding notes and bars let us exploit the long-range dependencies among the phrases using their spatial proximity. Intuitively, this is akin to building a mental map of a space explored by touch: though the touch information is sequential, the ing map can be two or three-dimensional, with significant amount of information encoded by proximity in the second and third dimensions. Traditionally, sequential data is learned using some variation of RNNs. Simple implementations of RNNs (standalone GRU and LSTM cells) tend to struggle with capturing long-distance dependencies; a popular solution to this issue is to use bi-directional RNNs (that is, a pair of layers that run in opposite directions) with an activation mechanism (using an auxiliary trainable network to better relate elements of the output sequence to those of the input sequence) so that information for multiple sections of the sequence can be captured . A similar effort at capturing long-distance dependencies has been applied to Convolutional Neural Networks (CNNs) too. Yu et al. have proposed dilated convolutions as a solution to the problem: in a dilated convolution, the convolution kernel is not a contiguous block, but rather subsections of the kernel are separated by some number of pixels from each other. As a , the convolution operation is not limited to the immediate neighbourhood, and long-distance dependencies are made available to the kernel . While attention-based and bi-directional RNNs have proven to be quite successful at capturing long-distance dependencies, we believe that there are a few notable benefits for being able to use a DCGAN for generating sequential data, as opposed to an RNN: • DCGANs that incorporate dilations allow for domain-specific heuristics to be passed to the neural network, by means of the dilation rates and the convolution kernel dimensions. This allows for a whole new dimension of flexibility and fine-tuning that is not available through RNNs. • A DCGAN yields both a discriminator and a generator, as opposed to just a generator. • Unlike an RNN, a GAN does not require a meaningful or domain-specific seed to be passed as an input to the generator; a vector of random noise can be used instead, making it easier to generate novel outputs from the generator. GANs have traditionally worked well with pictures, but to our knowledge existing music generation with GANs either relies on RNNs in the discriminator, as in SeqGAN or generates the melody in a sequential bar-after-bar process as in MidiNet . The discriminator starts off with a 6-tower convolution network (that is, 6 convolutions run sideby-side, as opposed to being dependent on each other). We chose to use dilated filters on 5 of the towers to capture relative information between bars and phrases (we shall refer to this as the global context), with the remaining layer being a contiguous 2x9 convolution to also capture the structure of the immediate neighbourhood (local context). Having multiple towers each learning different aspects gives us versatility and helps avoid the "tunnel vision" nature of other generative music models. This can be viewed in Figure 1. All the towers use 32 convolution filters, and use zero-padding for edge pixels that might not have a sufficient neighbourhood to be covered by each filter. The outputs of the convolutions from each tower are then stacked horizontally, and passed to a second (regular) layer of convolution made of 64 filters, using a 3x3 kernel, a 2x2 stride and with the convolution being only applied to pixels that have a sufficient neighbourhood for the convolution filter. The output of this layer is then flattened, and passed to a dense layer consisting of 1024 neurons, which is in turn followed by a sigmoid neuron that generates the prediction of the discriminator. No batch normalization is applied to any of the layers on the discriminator. Unlike the discriminator, the generator does not make use of dilations or towers. We start off with a dense layer made of 32x2x256 (where the last dimension is the number of filters). The dense layer is then reshaped into 256 filters of size 32x2, and passed through two layers of deconvolution, each with halving the number of filters, while using a stride of 1x1 and a kernel size of 2x5. A last deconvolution layer (also 2x5 in kernel size) is used to bring down the filters into one image, with a stride of 2x2 to return to the original image dimension of 64x4; this layer also uses a tanh activation to place the output values in the same range as the training dataset. All layers in the generator use batch normalization. After our GAN has generated music in the form of a 64x4 matrix, we map the real values to notes in ABC notation, rounding to the nearest note in the D major key. We then take each beat and merge notes with the same pitch together, so a sequence of four 16th notes C starting from a 1st, 5th, 9th or 13th note in a bar becomes a quarter note C instead. The ing ABC notated music can then be converted into sheet music or played with midi generators. Our approach is similar to the model by as both train on text-based musical notation and aim to learn Irish traditional instrumental music that has distinct structure. The models in Magenta are considered to be state of the art in music generation at the moment. For this reason, we compared our samples with the Folk-RNN and Magenta models -specifically the MelodyRNN model in Magenta -training using the same dataset of 820 curated reels. We generated an equivalent number of samples from each model and compared the samples. Since our goal is for the GAN to learn the global patterns, we use measures that highlight similarity in structure. The first metric we chose was the normalized mean Fréchet distance. Fréchet distance is a measure of similarity between vectors which prioritizes the ordering and magnitude of the values in those vectors. Since the order and distribution of notes are important for a tune to be considered structurally similar to another, this metric is ideal for us. We normalize this Fréchet distance so we can view the changes between the phrases better. The of this comparison are found in Figure 2. Each group records the normalized Fréchet distance between corresponding combinations of phrases i.e the first group highlights the normalized mean Fréchet distance between phrase 1 and 2 for all the distributions. We can see that our generated samples compare favorably with the Folk-RNN model and both exhibit similar structure to the training set. It is important to note that the MelodyRNN used here is the default model without the attention mechanism. Another measure to visualize similarity between distributions is the t-distributed stochastic neighbourhood embedding (t-SNE) algorithm developed byMaaten &. This algorithm groups points that are similar in higher dimension closer together in low dimensions -making it useful to visualize similarity between the tunes generated by both models along with the training data. Here, we set the tunable parameter perplexity to 50. This increases the importance of global similarity between vectors. Figure 3 shows distributions of the training data and samples generated by our model, FolkRNN and Magenta trained on the same data. Since t-SNE is stochastic and can generate different visualizations for each run, we display visualization produced by two runs of t-SNE algorithm. It can be observed that most of the time (>75%), the tunes generated by our model seem to lie within the distribution of the training data, although they only cover a subset of the training data. Both RNN models seem to mimic a slightly different distribution, where we can think of the tunes generated by Magenta as a subset of the FolkRNN ones, with FolkRNN generating by far the most diverse set of samples. A third metric is looking at the distribution of notes in the samples. Figure 4 shows the frequency of distribution of notes as midi key values in the samples. As music theory would suggest, the tonic of the key (in this case, the midi value corresponding to D) is the most common, with the 3rd and 5th of the key more frequent than the rest of the scale. Both our model and FolkRNN created a set of samples which mimics this distribution of the notes in the key, however Magenta seems to have a more uniform distribution of note frequencies. Overall, these metrics show that music generated by our DCGAN is comparable to melodies generated by RNNs on the same symbolic data. The use of dilations allows for our model to learn global tune structure. Converting sequential data into a format which implicitly encodes temporal information as spatial information is an effective way of generating samples of such data as whole pieces. Here, we explored this approach for melody generation of fixed-length music forms, such as an Irish reel, using non-recurrent architecture for the discriminator CNN with towers and dilations, as well as a CNN for the GAN itself. One advantage of this approach is that the model learns global and contextual information simultaneously, even with a small model. LSTMs and other approaches need a much larger model to be able to learn both the contextual neighboring note sequences and global melody structure. In future work, we would like to introduce boosting in order to capture the structure of the distribution more faithfully, and increase the range of pieces our model can generate. Natural extensions of the model would be to introduce multiple channels to capture durations better (for example, as in), and add polyphony (ie, using some form of piano roll representation). Another direction could be to experiment with higher-dimensional representation of the sequence data, to better capture several types of dependencies simultaneously. Additionally, it would be interesting to apply it to other kinds of fixed-length sequential data with long-range patterns.

Representing melodies as images with semantic units aligned we can generate them using a DCGAN without any recurrent components.

Represent the short summary for finding the corresponding scientific article (long)

We automatically extract fingering information from videos of piano performances, to be used in automatic fingering prediction models.


Question: Represent the scientific article to find a short TLDR

Question: Neural machine translation (NMT) systems have reached state of the art performance in translating text and widely deployed. Yet little is understood about how these systems function or break. Here we show that NMT systems are susceptible to producing highly pathological translations that are completely untethered from the source material, which we term hallucinations. Such pathological translations are problematic because they are are deeply disturbing of user trust and easy to find. We describe a method t generate hallucinations and show that many common variations of the NMT architecture are susceptible to them. We study a variety of approaches to reduce the frequency of hallucinations, including data augmentation, dynamical systems and regularization techniques and show that data augmentation significantly reduces hallucination frequency. Finally, we analyze networks that produce hallucinations and show signatures of hallucinations in the attention matrix and in the stability measures of the decoder. Neural machine translation (NMT) systems are language translation systems based on deep learning architectures BID10 BID1 BID31. In the past few years, NMT has vastly improved and has been deployed in production systems, for example at Google BID33, Facebook BID15, Microsoft BID17, and many others. As NMT systems are built on deep learning methodology, they exhibit both the strengths and weaknesses of the approach. For example, NMT systems are competitive with state of the art performance BID6 and scale well to very large datasets BID23 but like most large deep learning systems, NMT systems are poorly understood. For example, in many commercial translation systems, entering repeated words many times occasionally in strange translations, a phenomenon which has been highly publicized BID12. More broadly, recent work shows that NMT systems are highly sensitive to noise in the input tokens BID3 and also susceptible to adversarial inputs BID9. When there is an error in translation, it can be challenging to either understand why the mistake occurred or engineer a fix. Here we continue the study of noise in the input sequence and describe a type of phenomenon that is particularly pernicious, whereby inserting a single additional input token into the source sequence can completely divorce the translation from the input sentence. For example, here is a German input sentence translated to English (reference) by a small NMT system: Source: Caldiero sprach mit E! Nachrichten nach dem hart erkämpften Sieg, noch immer unter dem Schocküber den Gewinn des Großen Preises von 1 Million $. Reference: Caldiero spoke with E! News after the hard-fought victory, still in shock about winning the $1 million grand prize. NMT Translation: Caldiero spoke with E, after the hard won victory, still under the shock of the winning of the Grand Prix of 1 million $. Since its invention, researchers have been working to better understand NMT. For example, moving from the original Seq2Seq model BID31 BID11 to models that utilize attention mechanisms BID1, ed in improved translation quality BID22 and better interpretability BID14. Studies identified the most critical components of the LSTM BID16 and the role of the LSTM in language modeling BID19 more broadly. Followed by explorations in interpretability, recent work has focused on robust NMT, studying the effects of input noise, aiming to reduce variation from typos BID3 and synonym choice BID9 in the discrete input set used by NMT systems. Both BID3 and BID9 have discovered that NMT systems are highly sensitive to input noise and both used adversarial training to help stabilize NMT systems (either with black-box adversarial training or augmenting an adversarial loss). There has been work in understanding how to handle some of the pathologies in RNNs for language modeling and translation, for example, using scheduled sampling to handle mismatches between training and test sets BID5.In parallel, there has also been work in understanding RNNs through the framework of dynamical systems BID30 BID28 BID21 BID25 BID2 and understanding their capacity in language tasks BID13. For example, it is known that continuous time vanilla RNNs exhibit high-dimensional chaos BID27, which can be driven out by strong input BID24. RNNs exhibiting chaos can be beneficial, for example, RNNs capable of chaos in the absence of input can serve as strong initialization for training RNNs when they are input driven BID29, but caution must be used as unbridled chaos can be viewed as a source of dynamical noise. This has led to efforts to rid RNNs of chaos altogether BID21. Efforts related to improving optimization may also have dynamically regularizing effects, e.g. BID0. Given the complexities and slow compute times of recurrent systems, there have also been attempts to rid NMT of recurrence BID18 BID15 BID32.We further expand on these studies by highlighting the specific pathology of hallucinations, systematically studying those hallucinations, and analyzing them from a dynamical systems perspective. Models: In this paper, we use a standard RNN-based encoder-decoder NMT model with attention. Specifically, we study the NMT model described in BID33, known as GNMT. We use the GNMT model and its public implementation 2. Formally, given an input sequence, x 1:S of length S, the NMT model first encodes the input sequence x 1:S into a set of vectors z 1:S = f enc (x 1:S) using its encoder f enc. The task of the decoder f dec, is to generate the translation, y 1:T, one symbol at a time y i, given the encoding, z 1:S, and previously generated symbols y <i. The decoder, f dec is implemented as a conditional sequence model BID1, where the distribution over y 1:T is conditioned on x 1:S. The decoder internally makes use of an attention mechanism f att to query the encoder, summarizing z 1:S for each output symbol y i, putting it all together y i = f dec (y < i, f att (z 1:S)) (also see Figure 7 for a detailed model schematic). Finally, the conditional probability of the target sequence is modelled as p(y 1:T |x 1:S) = T i=1 p(y i |y < i, x 1:S) and the log of this conditional likelihood is maximized given a set of source-target pairs (x, y) during training. We study models that are significantly smaller and less complex than those typically used in state-ofthe-art or production systems for the sake of research tractability. We use a single layer bidirectional LSTM in the encoder f enc and a two layered unidirectional LSTM in the decoder f dec with an additive attention mechanism as f att BID8. The word embedding dimensions and each LSTM hidden cell (both in the encoder and decoder) are set to 256. We refer to this model as the canonical model. Unless otherwise stated, we used Adam BID20 optimizer with a learning rate of 0.001, a constant learning rate schedule, and clipped gradients to a maximum norm of 5.0 during training. Given these hyper-parameter and architectural choices, we trained 10 canonical models with different random seeds to observe how parameter initialization variability played a role in our . All additional model variants we describe later were also trained 10 times with the same 10 different random seeds. Each model was trained for 1M steps (updates) with a mini-batch of size 128 and the training checkpoint with the best BLEU score on the development set was selected. The central goal of our study was to understand how various modelling choices affected the frequency of hallucinations. In order to isolate the effects of modeling changes, all model variants we study in this paper were identical to the canonical model except for a single change. This means, for example, that our model with 512 hidden units also is 2 layers deep, etc. We performed a simple hyper-parameter search for the canonical model, and did not perform additional hyper-parameter searches for any additional models. All models we present are well trained with a BLEU score of at least 20.0 on the test set using greedy decoding, a reasonable score for 2-layer models with 256 hidden units. With beam search decoding, our canonical models achieve an average BLEU score of 25.66.Inference: Generating a translation of the input sequence, or formally finding an output sequence that maximizes the conditional log-probability,ŷ = argmax y log p(y|x), is a major challenge in NMT since the exact inference (or decoding) is intractable. NMT uses approximate decoding techniques which we also have used in this paper. The simplest approximate decoding technique, greedy decoding, chooses the most-likely symbol under the conditional probabilitŷ y t = argmax i log p(y t = i|ŷ <i, x 1:S), outputting a single best local prediction by keeping track of a single hypothesis k, at each time step. Another approximate decoding technique, beam search, improves upon greedy decoding by keeping track of multiple hypotheses (beams), where k >1 at each time step of the decoding, compared to k=1 in greedy-decoding. To maintain simplicity in our canonical model we used greedy decoding. Note that production systems will often perform beam search to find a more probable translation than one generated by greedy search. We also ran an additional set of experiments with beam search. We trained all models with the German to English WMT De→En 2016 dataset (4,500,966 examples) BID7, validated with the WMT De→En 2015 development set (2,169 examples). We then used the WMT De→En 2016 test set (2,999 examples) to compute the hallucination percentage for each model. For the input and output of all NMT models in consideration, we used sub-word tokens extracted by Byte-Pair Encoding (BPE) BID26. To construct a vocabulary of unique tokens, we first combined the tokenized source and target corpora BID2, and then learned a joint BPE code with an 8k merge operations budget, ing in 12,564 unique tokens. Further, in order to study the effect of larger vocabulary sizes, for some experiments we repeated the same process with 16,000 and 32,000 BPE codes and ended up with vocabularies having 19,708 and 36,548 unique tokens respectively. Note that we used the same vocabulary for both source and target side languages. Computing the percentage of hallucinations in a NMT model Select a model; Fix a random seed; Select a group of subword tokens with the following attributes:; -Common tokens: 100 most common subword tokens; -Mid-frequency tokens: random sample of 100 subword tokens between common and rare tokens; -Rare tokens: 100 least common subword tokens; -Punctuation: all punctuation tokens; for every sentence in test corpus (e.g. WMT De→En 2016 test set) do if adjusted BLEU between reference sentence and translated sentence > 0.09 then for every selected token do for every location in (beginning, end, second-to-end, randomly in the middle) do put the selected token at the selected location in the byte-pair encoded input sequence; translate the perturbed input sequence; if adjusted BLEU between the translated, perturbed sentence and the translated, unperturbed sentence< 0.01 then this sentence can be perturbed to produce a hallucination; Informally, a hallucination is a translation of a perturbed input sentence that has almost no words in common with the translation of the unperturbed sentence. Here, we use the term perturb to mean adding a single token to the input source sequence. This is based on an initial observation that adding a rare token to an input sequence reliably caused a model to generate a hallucination, e.g. adding a Chinese character token to a German to English translation. We expanded and systematized this discovery into a brute force search procedure (Algorithm 1) by splitting our tokens into several types: common (100 most common German tokens), rare (100 least common German tokens), mid-frequency tokens (randomly sampled 100 tokens from the remaining German tokens), and punctuation tokens. Additionally, we attempted to perturb each sentence by inserting a token at one of several positions: beginning, end, second to the end, and randomly in the middle. We did this for every sentence in our test set and collected statistics for each model variant. To define a quantitative threshold for a hallucination we modified the BLEU score, which is used to compare a reference sequence with a translated sequence. Briefly, the BLEU score is a common metric for translation, which measures weighted n-gram overlaps while penalizing short translations. We modified the BLEU score by re-weighting the n-grams in the BLEU score computation to favor having any words in common between the two sentences (1.0 for one-grams and 0.8 for bi-grams and disregarded other n-grams). Then, we call only sentences that have an adjusted BLEU score of less than 0.01 hallucinations. For examples of sentence pairs with different adjusted BLEU scores, see Section 8.2 in the Appendix. Not all translations are good even before adding a perturbation token to the input sequence. To strengthen our on hallucinations, we first excluded these poor translations by computing the adjusted BLEU score between the reference translation and the translation produced by the unperturbed input sequence. We kept sentences that had an adjusted BLEU of ≥ 0.09. We chose a value of 0.09 because it seemed to maintain enough context that you could tell the translation and the reference were related. We describe four common hallucination patterns: grammatically correct output that bears no relation to the input text, ungrammatical output with oscillatory structure, output that remains largely in the source language, and finally terse jumps to the end of the sequence. We also observe translations that are ungrammatical nonsense. While still highly undesirable, we note that a user should at least be able spot and reject these additional hallucination patterns. We show that hallucinations can be easily evoked by inserting tokens in the source sequence. We used Algorithm 1 to quantify how susceptible a given model is to hallucination. In particular, we studied what types of perturbations (location, and token type) are more effective at inducing hallucinations. With this method, we found that, on average, 73% of all sentences in the WMT De→En test set can be perturbed to hallucination in the canonical model. We studied how beam search, number of hidden units, vocabulary size, and decoding scheme affected hallucination percentages FIG0. We found that changing the number of hidden units to both 512 and 1024 from 256 and changing the vocabulary size-from 8K to 16K BPE codes did not significantly decrease the hallucination percentage. However, beam search and a vocabulary size increase corresponding to 32K BPE codes did significantly lower the mean percentage of hallucinations. We also studied how different types of perturbations impacted the hallucination percentage of the canonical model FIG0. By far, adding a perturbing token to the beginning of the input sequence induces the most hallucinations in the canonical model. We were curious if BLEU scores were predictive of hallucination percentage. We plot the BLEU vs. hallucination percentage of all models we study (FIG1 . Surprisingly, we found that hallucination percentage does not decrease as BLEU score increases. However, since we did not study all possible models, we urge caution in interpreting these . What changes can we make to the model to make it more robust to hallucinations? We investigated the effect of three different methodologies, simple regularizations, data augmentation and regularizations on the dynamics in state space. We tested if a model variation significantly reduces hallucinations by performing a one-sided Mann-Whitney U between the canonical distribution of models and the distribution of models that use the model variant. We use a p-value of 0.05.Simple Regularizations: We choose dropout, L2 regularization on embeddings (L2E), L2 regularization on recurrent weights (L2R) and L2 regularization on all weights (L2) as straight-forward regularization techniques to be applied. For dropout, we created a model with dropout in all feedforward layers, with a keep probability of 0.9. Next, we implemented L2 regularization on the We augmented the training data by perturbing all training sentences with a random token (either common, rare, mid-frequency, or punctuation) at either the beginning, end, second-to-end, or randomly in the middle while keeping the reference translation the same. This doubled our training set. We then trained a canonical model with the augmented training set, and found that data augmentation helps decrease hallucination percentages. We call this model DA.Dynamical Regularizations: We wondered if hallucinations could be reduced by providing a more exact initial state for the decoder, so we trained additional models where the initial state of the decoder was tied to last step of the encoder (TDIS). Note that the canonical model sets the decoder initial state as a vector of zeros. As a second regularization method that operates on the state space, we used Chaos-free network (CFN) BID21 which by premise cannot produce chaos. We replaced the LSTM cell with the CFN in a set of experiments, again using 256 hidden units. Dropout, L2E, and DA all ed in statistically significant decreases in hallucination percentage, with DA being by far the most effective at decreasing hallucinations. On the contrary, switching out Although data augmentation dramatically reduced hallucinations in the canonical model, it requires knowing the kind of perturbations that one would use to induce a hallucination. To study how fine grained one's knowledge must be, we trained the canonical model on a different training set where we withheld two types of data augmentation: perturbing at the beginning or with common tokens (We call this model DA w/o beginning or common). We then compared this model with the canonical model trained with the full DA training set (Figure 4). We found that DA w/o beginning or common yields much higher hallucination percentages when tested by perturbing at the beginning or with common tokens in comparison to the DA model. However, we also saw a reduction in hallucination percentage for common and beginning tokens when compared to the canonical model. This indicates that DA can still provide some protection against hallucinations, even if exact perturbations are not known. Figure 4: Effects of data augmented (DA) training when including all perturbation types vs excluding common and beginning perturbation types. We trained two models, one including all perturbations types for DA training, and the other excluding common and beginning perturbation types. We then examined the hallucination percentage of each perturbation type for both of these models and studied whether a DA model would be less prone to hallucinate when perturbed with types of tokens or positions it had not been trained against. Red star shows that DA w/o beginning or common had statistically significantly reduced mean compared to the canonical model trained without DA.Additionally, we wondered if hallucinations were present in NMT architectures that were not recurrent. Thus, we studied the Transformer model (TR) BID32. To make our easily accessible to NMT practitioners, we chose a hyperparameter set from those given in the popular Tensor2Tensor library BID3 that was closest to our cutoff BLEU score when using greedy decoding. These models are trained with the parameters from transformer tiny (2 hidden layers, 128 hidden size, 512 filter size, and 4 heads) and have a greedy BLEU score of 17.5, which is a little lower than our GNMT models. We find the transformer model hallucinates significantly less than the canonical model, but can still be perturbed to hallucinate on average 15% of the time FIG2 ). We present these with many caveats. Unlike the canonical model, this model is trained with many types of regularization (dropout, attention dropout, label smoothing, relu dropout, and a larger batch size) and a longer input sequence length (256 versus 50 in the canonical model). Unfortunately, training with no regularization or a sequence length of 50 dramatically reduced the BLEU score for parameter combinations we tried, and thus we decided to present these with caveats instead of a model without regularization and a comparable sequence length. We observed a large difference between attention matrices of normal translations and of hallucinations. Attention networks in normal translations tend to study the entire input sequence throughout decoding. In French to English and other language pairs that are grammatically aligned (German to English is somewhat aligned), this often in a strong diagonal in the attention matrix. The attention matrix, when translating hallucinations, however, shows the model attending to a few tokens. We give an example comparison in FIG3, top panels. For additional attention matrices, see Section 9.We wanted to quantify this difference in a way that does not require strong alignment between languages, i.e. one expects English to French to in a largely diagonal matrix but not English to Turkish, so we used information entropy to compute a statistic that described the difference between attention matrices during a decode that ed in a hallucination and those that ed in a normal translation. Specifically, at each output of the decoder, the attention network gives a distribution over input tokens. We averaged these distributions across all decoded output tokens, ing in a distribution of average attention weight over the input tokens. We treated this as a discrete distribution and computed the entropy, − t p(x t) log p(x t), where x t is the input token at time t, for each example, ing in a distribution of entropy values over all decoded sequences. We then compared the entropy of average attention distributions between hallucinations and correct translations (FIG3 . This figure shows a significant difference between the entropy values for hallucination sequences. As a control, we show there is no significant difference between original input sequences and perturbed input sequences for sentences that cannot be perturbed to hallucination. Note that, in real world scenarios where a ground truth translation is not available, the measure of entropy of the average attention distribution may be useful to detect hallucinations. The breakdown of the attention module seems to signal that the encoder and the decoder have been decoupled, and the decoder ignores context from the encoder and samples from its language model. Two possibilities are that broken attention modules are the root cause of decoupling, or they are a symptom of further breakdown in the dynamics of the decoder. Examining the causes and types of hallucinations: instability of translation, translations decoupled from the input, and oscillations in translations, led us to believe that hallucinations from a dynamical process gone wrong in the decoder (which might be caused by the encoder or attention module). Further, many model variants that decrease hallucinations (like L2 regularization) can be viewed as regularizing the dynamics of the model. Thus, we explore differences in the decoder between translating a hallucinating or non-hallucinating sentence by comparing the hidden states of the decoder and analyzing the stability of the system FIG4 ).In this section, we are interested in how perturbations change the decoding pipeline and thus study an unchanged, input sentence (which we call "original" and denote by x o) and its perturbations (x p). We perturbed all source sentences in the test set with all types of tokens (common, rare, mid-frequency, and punctuation) at all locations (beginning, end, second-to-end, and randomly) and sorted all perturbations into two groups: those that in hallucinations and those that do not. We hypothesize that differences in translation occur early in decoding and focus on the first timestep the decoder receives context from the encoder, t = 1. We studied both the distance between (left panel) and ratio of the norms (middle panel) of the perturbed decoder hidden states h 1 (x p) and the original decode, h 1 (x o), at decode t = 1. Both ed in obviously different distributions as a function of whether the perturbation ed in a hallucination. Given these differences (FIG4), it seemed natural to attempt to causally reduce the number of perturbations by re-normalizing h 1 (x p) to the norm of h 1 (x o). We tried this for all perturbations of all original sentences in our test set and did not see a reduction in hallucination percentage. We wondered if a hallucination from exciting an unstable mode of the decoder early in the decode. To study this, we defined an unstable subspace of the original decode U (x o) as the subspace spanned by the eigenvectors of DISPLAYFORM0 ∂h0 (x o) with corresponding eigenvalues greater than 1. We projected the normalized hidden state of the perturbed input, x p, onto this subspace A. Normal translations produce attention matrices that show a distribution of weight across most tokens in the source sequence throughout decoding (x-axis source sequence, y-axis decoded sequence). B. However, during hallucinations, the attention network tends to place weight on only a few input tokens. We can see the majority of the weight throughout decoding is placed on the "." in the source sequence. C, D. We used an entropy measure to quantify how distributed the attention is over the input source sequence. Shown are the distributions for normal translations C. and hallucinations D. for the original (blue) and perturbed (green) attention entropy values. Mean of the entropy distributions for hallucinations are statistically significantly different (Mann-Whitney U test, p < 0.05). DISPLAYFORM1 whereĥ is h normalized to 1. We did this for every original sentence in our test set that had at least 10 hallucinations (around half of all original sentences). We show the count (up to 10) of perturbed sentences such that E(x o, x p) > 1 when x p ed in a hallucination (red) and when it did not (blue) in FIG4 (right panel).Finally, we also studied the stability exponents of the decoder, focusing on how eigenvalues of the Jacobian, ∂h T ∂h0 (x), of the hidden states of the decoder changed as a function of whether or not a perturbation ed in a hallucination for models trained with and without data augmentation (Shown in Appendix 10). In this paper we uncovered and studied a hallucination-like phenomenon whereby adding a single additional token into the input sequence causes complete mistranslation. We showed that hallucinations are common in the NMT architecture we examined, as well as in its variants. We note that hallucinations appear to be model specific. We showed that the attention matrices associated with hallucinations were statistically different on average than those associated with input sentences that could not be perturbed. Finally we proposed a few methods to reduce the occurrence of hallucinations. Our model has two differences from production systems. For practical reasons we studied a small model and used a limited amount of training data. Given these differences it is likely that our model shows more hallucinations than a quality production model. However, given news reports of strange translations in popular public translation systems BID12, the dynamical nature of the phenomenon, the fact that input datasets are noisy and finite, and that our most effective technique for preventing hallucinations is a data augmentation technique that requires knowledge of hallucinations, it would be surprising to discover that hallucinations did not occur in production systems. While it is not entirely clear what should happen when a perturbing input token is added to an input source sequence, it seems clear that having an utterly incorrect translation is not desirable. This phenomenon appeared to us like a dynamical problem. Here are two speculative hypotheses: perhaps a small problem in the decoder is amplified via iteration into a much larger problem. Alternatively, perhaps the perturbing token places the decoder state in a poorly trained part of state space, the dynamics jump around wildly for while until an essentially random well-trodden stable trajectory is found, producing the remaining intelligible sentence fragment. Many of our can be interpreted from the vantage of dynamical systems as well. For example, we note that the NMT networks using CFN recurrent modules were highly susceptible to perturbations in our experiments. This highlights the difficulty of understanding or fixing problems in recurrent networks. Because the CFN is embedded in a larger graph that contains an auto-regressive loop, there is no guarantee that the chaos-free property of the CFN will transfer to the larger graph. The techniques we used to reduce hallucinations can also be interpreted as dynamical regularization. For example, L2 weight decay is often discussed in the context of generalization. However, for RNNs L2 regularization can also be thought of as dynamically conditioning a network to be more stable. L2 regularization of input embeddings likely means that rare tokens will have optimization pressure to reduce the norm of those embeddings. Thus, when rare tokens are inserted into an input token sequence, the effects may be reduced. Even the data augmentation technique appears to have stability effects, as Appendix 10 shows the overall stability exponents are reduced when data augmentation is used. Given our experimental , do we have any recommendations for those that engineer and maintain production NMT systems? Production models should be tested for hallucinations, and when possible, the attention matrices and hidden states of the decoder should be monitored. Our on reducing hallucinations suggest that standard regularization techniques such as Dropout and L2 weight decay on the embeddings are important. Further, data augmentation seems critical and we recommend inserting randomly chosen perturbative tokens in the input sentence as a part of the standard training regime (while monitoring that the BLEU score does not fall). We note a downside of data augmentation is that, to some extent, it requires knowing the types of the pathological phenomenon one desires to train against. Figure 7: Schematic of the NMT decoder. The input sequence, x 1:S, is encoded by a bidirectional encoder (not shown) into a sequence of encodings, z 1:S. The attention network, f att, computes a weighted sum of these encodings (computed weights not shown), based on conditioning information from h and provides the weighted encoding to the 2-layer decoder, f dec, as indicated by the arrows. The decoder proceeds forward in time producing the translation one step at a time. As the decoder proceeds forward, it interacts with both the attention network and also receives as input the decoded output symbol from the previous time step. Examples of pairs of sentences with different adjusted BLEU scores are as follows: As seen above, an adjusted BLEU score of < 0.01 means the two sentences have very few words in common. DISPLAYFORM0 We defined a spectrum of stability exponents for the decoder and compared them between normal translations and hallucinations (FIG6). Concretely, we studied the stability of the decoder as a function of a given input token sequence, x 1:S of length S (denoted x below). The sequence x 1:S is run through the encoder, whose output is processed by the attention network, finally delivering an input to the decoder. For a given input token sequence, the decoder runs until it produces an endof-sequence token, ing in an output token sequence y 1:T of length T (or reaches a maximal decoded sequence length T > 3S). We were interested in studying DISPLAYFORM0 ∂h0 as many stability properties can be deduced from it. We note that if one is interested in studying ∂ht ∂xs, the iterative process described by ∂h T ∂h0 would still be critical to understand due to the chain rule. We defined our spectrum of stability exponents, in analogy with Lyapunov exponents, but adapted for finite time by studying a finite-time version of Oseledets matrix, typically used in the study of chaotic dynamical systems. In particular, the i th stability exponent is defined as λ i (x) = 1 2T log (α i (x)), where α i (x) is the i th eigenvalue of the positive-semidefinite symmetric matrix DISPLAYFORM1 ∂h0 (x) and h(t) is the decoder state at time t concatenated across all layers. We used auto-differentiation software to exactly compute the Jacobian ∂h T ∂h0 (x) so the complexities of the decoder circuitry were handled naturally (shown in Appendix, Section 8.1 BID4 .We show the distribution of stability exponents, comparing between all input sequences that could be made to hallucinate, and all those that could not ( FIG6). We show these for both the canonical model and the model trained with data augmentation. There are two observations. First, means of the distribution of the stability exponents for the canonical model, averaged over those sentences that could be perturbed to hallucinate, are statistically different than exponents averaged over sentences that could not be perturbed to hallucinate. Second, the distributions of the model trained with data augmentation show significantly reduced exponents in comparison to the canonical model.

Represent the TLDR for finding the corresponding scientific article

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.

We introduce MIST RNNs, which a) exhibit superior vanishing-gradient properties in comparison to LSTM; b) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies; and c) are much more efficient than previously-proposed NARX RNNs, with even fewer parameters and operations than LSTM.


Question: Represent the article for finding a single-sentence TLDR

Question: For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks. In this work, we identify two issues of current explanatory methods. First, we show that two prevalent perspectives on explanations—feature-additivity and feature-selection—lead to fundamentally different instance-wise explanations. In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals. The second issue is that current post-hoc explainers have only been thoroughly validated on simple models, such as linear regression, and, when applied to real-world neural networks, explainers are commonly evaluated under the assumption that the learned models behave reasonably. However, neural networks often rely on unreasonable correlations, even when producing correct decisions. We introduce a verification framework for explanatory methods under the feature-selection perspective. Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able to provide guarantees on its inner workings. We validate the efficacy of our evaluation by showing the failure modes of current explainers. We aim for this framework to provide a publicly available,1 off-the-shelf evaluation when the feature-selection perspective on explanations is needed. A large number of post-hoc explanatory methods have recently been developed with the goal of shedding light on highly accurate, yet black-box machine learning models (a; ; ; ; b; ;). Among these methods, there are currently at least two widely used perspectives on explanations: feature-additivity (a; ; ;) and feature-selection (; ;), which we describe in detail in the sections below. While both shed light on the overall behavior of a model, we show that, when it comes to explaining the prediction on a single input in isolation, i.e., instance-wise explanations, the two perspectives lead to fundamentally different explanations. In practice, explanatory methods adhering to different perspectives are being directly compared. For example, and compare L2X, a feature-selection explainer, with LIME (a) and SHAP , two feature-additivity explainers. We draw attention to the fact that these comparisons may not be coherent, given the fundamentally different explanation targets, and we discuss the strengths and limitations of the two perspectives. Secondly, while current explanatory methods are successful in pointing out catastrophic biases, such as relying on headers to discriminate between pieces of text about Christianity and atheism (a), it is an open question to what extent they are reliable when the model that they aim to explain (which we call the target model) has a less dramatic bias. This is a difficult task, precisely because the ground-truth decision-making process of neural networks is not known. Consequently, when applied to complex neural networks trained on real-world datasets, a prevalent way to evaluate the explainers is to assume that the target models behave reasonably, i.e., that they did not rely on irrelevant correlations. For example, in their morphosyntactic agreement paradigm, Pörner et al. assume that a model that predicts if a verb should be singular or plural given the tokens before the verb, must be doing so by focusing on a noun that the model had identified as the subject. Such assumptions may be poor, since recent works show a series of surprising spurious correlations in human-annotated datasets, on which neural networks learn to heavily rely (; ;). Therefore, it is not reliable to penalize an explainer for pointing to tokens that just do not appear significant to us. We address the above issue by proposing a framework capable of generating evaluation tests for the explanatory methods under the feature-selection perspective. Our tests consist of pairs of (target model, dataset). Given a pair, for each instance in the dataset, the specific architecture of our model allows us to identify a subset of tokens that have zero contribution to the model's prediction on the instance. We further identify a subset of tokens clearly relevant to the prediction. Hence, we test if explainers rank zero-contribution tokens higher than relevant tokens. We instantiated our framework on three pairs of (target model, dataset) on the task of multi-aspect sentiment analysis. Each pair corresponds to an aspect and the three models (of same architecture) have been trained independently. We highlight that our test is not a sufficient test for concluding the power of explainers in full generality, since we do not know the whole ground-truth behaviour of the target models. Indeed, we do not introduce an explanation generation framework but a framework for generating evaluation tests for which we provide certain guarantees on the behaviour of the target model. Under these guarantees we are able to test the explainers for critical failures. Our framework therefore generates necessary evaluation tests, and our metrics penalize explainers only when we are able to guarantee that they produced an error. To our knowledge, we are the first to introduce an automatic and non-trivial evaluation test that does not rely on speculations on the behavior of the target model. Finally, we evaluate L2X , a feature-selection explainer, under our test. Even though our test is specifically designed for feature-selection explanatory methods, since, in practice, the two types of explainers are being compared, and, since LIME (a) and SHAP are two very popular explainers, we were interested in how the latter perform on our test, even though they adhere to the feature-additivity perspective. Interestingly, we find that, most of the time, LIME and SHAP perform better than L2X. We will detail in Section 5 the reasons why we believe this is the case. We provide the error rates of these explanatory methods to raise awareness of their possible modes of failure under the feature-selection perspective of explanations. For example, our findings show that, in certain cases, the explainers predict the most relevant token to be among the tokens with zero contribution. We will release our test, which can be used off-the-shelf, and encourage the community to use it for testing future work on explanatory methods under the feature-selection perspective. We also note that our methodology for creating this evaluation is generic and can be instantiated on other tasks or areas of research. The most common instance-wise explanatory methods are feature-based, i.e., they explain a prediction in terms of the input unit-features (e.g., tokens for text and super-pixels for images). Among the feature-based explainers, there are two major types of explanations: (i) feature-additive: provide signed weights for each input feature, proportional to the contributions of the features to the model's prediction (a; ; ;), and (ii) feature-selective: provide a (potentially ranked) subset of features responsible for the prediction (; ;). We discuss these explanatory methods in more detail in Section 3. Other types of explanations are (iii) example-based : identify the most relevant instances in the training set that influenced the model's prediction on the current input, and (iv) human-level explanations (; ;): explanations that are similar to what humans provide in real-world, both in terms of arguments (human-biases) and form (full-sentence natural language). In this work, we focus on verifying feature-based explainers, since they represent the majority of current works. While many explainers have been proposed, it is still an open question how to thoroughly validate their faithfulness to the target model. There are four types of evaluations commonly performed: 1. Interpretable target models. Typically, explainers are tested on linear regression and decision trees (e.g., LIME (a)) or support vector representations (e.g., MAPLE ). While this evaluation accurately assesses the faithfulness of the explainer to the target model, these very simple models may not be representative for the large and intricate neural networks used in practice. 2. Synthetic setups. Another popular evaluation setup is to create synthetic tasks where the set of important features is controlled. For example, L2X was evaluated on four synthetic tasks: 2-dim XOR, orange skin, nonlinear additive model, and switch feature. While there is no limit on the complexity of the target models trained on these setups, their synthetic nature may still prompt the target models to learn simpler functions than the ones needed for real-world applications. This, in turn, may ease the job for the explainers. 3. Assuming a reasonable behavior. In this setup, one identifies certain intuitive heuristics that a high-performing target model is assumed to follow. For example, in sentiment analysis, the model is supposed to rely on adjectives and adverbs in agreement with the predicted sentiment. Crowd-sourcing evaluation is often performed to assert if the features produced by the explainer are in agreement with the model's prediction . However, neural networks may discover surprising artifacts to rely on, even when they obtain a high accuracy. Hence, this evaluation is not reliable for assessing the faithfulness of the explainer to the target model. 4. Are explanations helping humans to predict the model's behaviour? In this evaluation, humans are presented with a series of predictions of a model and explanations from different explainers, and are asked to infer the predictions (outputs) that the model will make on a separate set of examples. One concludes that an explainer E1 is better than an explainer E2 if humans are consistently better at predicting the output of the model after seeing explanations from E1 than after seeing explanations from E2 . While this framework is a good proxy for evaluating the real-world usage of explanations, it is expensive and requires considerable human effort if it is to be applied on complex real-world neural network models. In contrast to the above, our evaluation is fully automatic, the target model is a non-trivial neural network trained on a real-world task and for which we provide guarantees on its inner-workings. Our framework is similar in scope with the sanity check introduced by. However, their test filters for the basic requirement that an explainer should provide different explanations for a model trained on real data than when the data and/or model are randomized. Our test is therefore more challenging and requires a stronger fidelity of the explainer to the target model. As mentioned before, current explanatory methods adhere to two major perspectives of explanations: Perspective 1 (Feature-additivity): For a model f and an instance x, the explanation of the prediction f (x) consists of a set of contributions {w x i (f)} i for each feature i of x such that the sum of the contributions of the features in x approximates f (x), i.e., i w Many explanatory methods adhere to this perspective (; ; a). For example, LIME (a) learns the weights via a linear regression on the neighborhood (explained below) of the instance. unified this class of methods by showing that the only set of feature-additive contributions that verify three desired constraints (local accuracy, missingness, and consistency-we refer to their paper for details) are given by the Shapley values from game theory: where the sum enumerates over all subsets x of features in x that do not include the feature i, and | · | denotes the number of features of its argument. Thus, the contribution of each feature i in the instance x is an average of its contributions over a neighborhood of the instance. Usually, this neighborhood consists of all the perturbations given by x 1: "The movie was good, it was actually x 2: "The movie was nice, in fact, it was nice. " very good." masking out combinations of features in x; see, e.g., (a;). show that the choice of the neighborhood is critical, and it is an open question what neighborhood is best to use in practice. Perspective 2 (Feature-selection): For a model f and an instance x, the explanation of f (x) consists of a sufficient (ideally small) subset S(x) of (potentially ranked) features that alone lead to (almost) the same prediction as the original one, i.e., f (S(x)) ≈ f (x).; , and adhere to this perspective. For example, L2X learns S(x) by maximizing the mutual information between S(x) and the prediction. However, it assumes that the number of important features per instance, i.e., |S(x)|, is known, which is usually not the case in practice. A downside of this perspective is that it may not always be true that the model relied only on a (small) subset of features, as opposed to using all the features. However, this can be the case for certain tasks, such as sentiment analysis. To better understand the differences between the two perspectives, in Figure 1, we provide the instance-wise explanations that each perspective aims to provide for a hypothetical sentiment analysis regression model, where 0 is the most negative and 1 the most positive score. We note that our hypothetical model is not far, in behaviour, from what real-world neural networks learn, especially given the notorious biases in the datasets. For example, show that natural language inference neural networks trained on SNLI may heavily rely on the presence of a few specific tokens in the input, which should not even be, in general, indicators for the correct target class, e.g., "outdoors" for the entailment class, "tall" for the neutral class, and "sleeping" for the contradiction class. In our examples in Figure 1, we clearly see the differences between the two perspectives. For the instance x 1, the feature-additive explanation tells us that "nice" was the most relevant feature, with a weight of 0.4, but also that "good" had a significant contribution of 0.3. While for this instance alone, our model relied only on "nice" to provide a positive score of 0.7, it is also true that, if "nice" was not present, the model would have relied on "good" to provide a score of 0.6. Thus, we see that the feature-additive perspective aims to provide an average explanation of the model on a neighborhood of the instance, while the feature-selective perspective aims to tell us the pointwise features used by the model on the instance in isolation, such as "nice" for instance x 1. An even more pronounced difference between the two perspectives is visible on instance x 2, where the ranking of features is now different. The feature-selective explanation ranks "good" and "nice" as the two most important features, while on the instance x 2 in isolation, the model relied on the tokens "very" and "good", that the feature-selection perspective would aim to provide. Therefore, we see that, while both perspectives of explanations give insights into the model's behavior, one perspective might be preferred over the other in different real-world use-cases. In the rest of the paper, we propose a verification framework for the feature-selection perspective of instance-wise explanations. Our proposed verification framework leverages the architecture of the RCNN model introduced by. We further prune the original dataset on which the RCNN had been trained to ensure that, for each datapoint x, there exists a set of tokens that have zero contribution (irrelevant features) and a set of tokens that have a significant contribution (clearly relevant features) to RCNN's prediction on x. We further introduce a set of metrics that measure how explainers fail to rank the irrelevant tokens lower than the clearly revelant ones. We describe each of these steps in detail below. The RCNN. The RCNN consists of two modules: a generator followed by an encoder, both instantiated with recurrent convolutional neural networks . The generator is a bidirectional network that takes as input a piece of text x and, for each of its tokens, outputs the parameter of a Bernoulli distribution. According to this distribution, the RCNN selects a subset of tokens from x, called S x = generator(x), and passes it to the encoder, which makes the final prediction solely as a function of S x. Thus: There is no direct supervision on the subset selection, and the generator and encoder were trained jointly, with supervision only on the final prediction. The authors also used two regularizers: one to encourage the generator to select a short sub-phrase, rather than disconnected tokens, and a second to encourage the selection of fewer tokens. At training time, to circumvent the non-differentiability introduced by the intermediate sampling, the gradients for the generator were estimated using a REINFORCE-style procedure . This intermediate hard selection facilitates the existence of tokens that do not have any contribution to the final prediction. aimed for S x to be the sufficient rationals for each prediction, the model might have learned an internal (emergent) communication protocol that encodes information from the non-selected via the selected tokens, which we call a handshake. For example, the RCNN could learn a handshake such as the one in Figure 2, where the feature "good" was important in all three cases, but not selected in the first two. Eliminating handshakes. Our goal is to gather a dataset D such that for all x ∈ D, the set of non-selected tokens, which we denote N x = x \ S x, has zero contribution to the RCNN's prediction on x. Equivalently, we want to eliminate instances that contain handshakes. We show that: The proof is in Appendix B. On our example in Figure 2, on the instance "The movie was very good.", the model selects "very" and predicts a score of 1. However, if we input the instance consisting of just "very", the model will not select anything 2 and would return a score of 0.5. Thus, Equation 7 indeed captures the handshake in this example. From now on, we refer to non-selected tokens as irrelevant or zero-contribution interchangeably. On the other hand, we note that S Sx = S x does not necessarily imply that there was a handshake. There might be tokens (e.g., the or a at the ends of the selection sequence(s)) that might have been selected in the original instance x and that become non-selected in the instance formed by S x without significantly changing the actual prediction. However, since it would be difficult to differentiate between such a case and an actual handshake, we simply prune the dataset by retaining only the instances for which S Sx = S x. At least one clearly relevant feature. With our pruning above, we ensured that the non-selected tokens have no contribution to the prediction. However, we are yet not sure that all the non-selected tokens are relevant to the prediction. In fact, it is possible that some tokens (such as "the" or "a") are actually noise, but have been selected only to ensure that the selection is a contiguous sequence (as we mentioned, the RCNN was penalized during training for selecting disconnected tokens). Since we do not want to penalize explainers for not differentiating between noise and zero-contribution features, we further prune the dataset such that there exists at least one selected token which is, without any doubt, clearly relevant for the prediction. To ensure that a given selected token s is clearly relevant, we check that, when removing the token s, the absolute change in prediction with respect to the original prediction is higher than a significant threshold τ. Precisely, if for the selected token s ∈ S x, we have that |encoder(S x −s)−encoder(S x)| ≥ τ, then the selected token s is clearly relevant for the prediction. Thus, we have further partitioned S x into S x = SR x ∪ SDK x, where SR x are the clearly relevant tokens, and SDK x are the rest of the selected tokens for which we do not know if they are relevant or noise (SDK stands for "selected don't know"). We see a diagram of this partition in Figure 3. We highlight that simply because a selected token alone did not make a change in prediction higher than a threshold does not mean that this token is not relevant, as it may be essential in combination with other tokens. Our procedure only ensures that the tokens that change the prediction by a given (high) threshold are indeed important and should therefore be ranked higher than any of the non-selected tokens, which have zero contribution. We thus further prune the dataset to retain only the datapoints x for which |SR x | ≥ 1, i.e., there is at least one clearly relevant token per instance. Evaluation metrics. First, we note that our procedure does not provide an explainer in itself, since we do not give an actual ranking, nor any contribution weights, and it is possible for some of the tokens in SDK x to be even more important than tokens in SR x. However, we guarantee the following two properties: P1: All tokens in N x have to be ranked lower than any token in SR x. The first most important token has to be in S x. We evaluate explainers that provide a ranking over the features. We denote by r 1 (x), r 2 (x),..., r n (x) the ranking (in decreasing order of importance) given by an explainer on the n = |x| features in the instance x. Under our two properties above, we define the following error metrics: (A) Percentage of instances for which the most important token provided by the explainer is among the non-selected tokens: where 1 is the indicator function. (B) Percentage of instances for which at least one non-selected token is ranked higher than a clearly relevant token: (C) Average number of non-selected tokens ranked higher than any clearly relevant token: where last si = argmax j {r j (x) ∈ SR x } is the lowest rank of the clearly relevant tokens. Metric (A) shows the most dramatic failure: the percentage of times when the explainer tells us that the most relevant token is one of the zero contribution ones. Metric (B) shows the percentage of instances for which there is at least an error in the explanation. Finally, metric (C) quantifies the number of zero-contribution features that were ranked higher than any clearly relevant feature. In this work, we instantiate our framework on the RCNN model trained on the BeerAdvocate corpus, 3 on which the RCNN was initially evaluated . BeerAdvocate consists of a total of ≈.100K human-generated multi-aspect beer reviews, where the three considered aspects are appearance, aroma, and palate. The reviews are accompanied with fractional ratings originally between 0 and 5 for each aspect independently. The RCNN is a regression model with the goal to predict the rating, rescaled between 0 and 1 for simplicity. Three separate RCNNs are trained, one for each aspect independently, with the same default settings. 4 With the above procedure, we gathered three datasets D a, one for each aspect a. For each dataset, we know that for each instance x ∈ D a, the set of non-selected tokens N x has zero contribution to the prediction of the model. For obtaining the clearly relevant tokens, we chose a threshold of τ = 0.1, since the scores are in, and the ground-truth ratings correspond to {0, 0.1, 0.2, . . ., 1}. Therefore, a change in prediction of 0.1 is to be considered clearly significant for this task. We provide several statistics of our datasets in Appendix A. For example, we provide the average lengths of the reviews, of the selected tokens per review, of the clearly relevant tokens among the selected, and of the non-selected tokens. We note that we usually obtained 1 or 2 clearly relevant tokens per datapoints, showing that our threshold of 0.1 is likely very strict. However, we prefer to be more conservative in order to ensure high guarantees on our evaluation test. We also provide the percentages of datapoints eliminated in order to ensure the no-handshake condition (Equation 7). Evaluating explainers. We test three popular explainers: LIME (a), SHAP , and L2X . We used the code of the explainers as provided in the original repositories, 5 with their default settings for text explanations, with the exception that, for L2X, we set the dimension of the word embeddings to 200 (the same as in the RCNN), and we also allowed training for a maximum of 30 epochs instead of 5. As mentioned in Section 3, LIME and SHAP adhere to the feature-additivity perspective, hence our evaluation is not directly targeting these explainers. However, we see in Table 1 that, in practice, LIME and SHAP outperformed L2X on the majority of the metrics, even though L2X is a featureselection explainer. We hypothesize that a major limitation of L2X is the requirement to know the number of important features per instance. Indeed, L2X learns a distribution over the set of features by maximizing the mutual information between subsets of K features and the response variable, where K is assumed to be known. In practice, one usually does not know how many features per instance a model relied on. To test L2X under real-world circumstances, we used as K the average number of tokens highlighted by human annotators on the subset manually annotated by. We obtained an average K of 23, 18, and 13 for the three aspects, respectively. In Table 1, we see that, on metric (A), all explainers are prone to stating that the most relevant feature is a token with zero contribution, as much as 14.79% of the time for LIME and 12.95% of the time for L2X in the aroma aspect. We consider this the most dramatic form of failure. Metric (B) shows that both explainers can rank at least one zero-contribution token higher than a clearly relevant feature, i.e., there is at least one mistake in the predicted ranking. Finally, metric (C) shows that, in average, SHAP only places one zero-contribution token ahead of a clearly relevant token for the first two aspects and around 9 tokens for the third aspect, while L2X places around 3-4 zero-contribution tokens ahead of a clearly relevant one for all three aspects. Figure 4: Explainers' rankings (with top 5 features on the right-hand side) on an instance from the palate aspect in our evaluation dataset. Qualitative Analysis. In Figure 6, we present an example from our dataset of the palate aspect. More examples in Appendix C. The heatmap corresponds to the ranking determined by each explainer, and the intensity of the color decreases linearly with the ranking of the tokens. 6 We only show in the heatmap the first K = 10 ranked tokens, for visibility reasons. Tokens in S x are in bold, and the clearly relevant tokens from SR x are additionally underlined. The first selected by the explainer is marked wth a rectangular. Additionally the 5 ranks tokens by each explainer are on the right-hand side. Firstly, we notice that both explainers are prone to attributing importance to nonselected tokens, with LIME and SHAP even ranking the tokens "mouthfeel" and "lacing" belonging to N x as first two (most important). Further, "gorgeous", the only relevant word used by the model, did not even make it in top 13 tokens for L2X. Instead, L2X gives "taste", "great", "mouthfeel" and "lacing" as most important tokens. We note that if the explainer was evaluated by humans assuming that the RCNN behaves reasonably, then this choice could have well been considered correct. In this work, we first shed light on an important distinction between two widely used perspectives of explanations. Secondly, we introduced an off-the-shelf evaluation test for post-hoc explanatory methods under the feature-selection perspective. To our knowledge, this is the first automatic verification framework offering guarantees on the behaviour of a non-trivial real-world neural network. We presented the error rates on different metrics for three popular explanatory methods to raise awareness of the types of failures that these explainers can produce, such as incorrectly predicting even the most relevant token. While instantiated on a natural language processing task, our methodology is generic and can be adapted to other tasks and other areas. For example, in computer vision, one could train a neural network that first makes a hard selection of super-pixels to retain, and subsequently makes a prediction based on the image where the non-selected super-pixels have been blurred. The same procedure of checking for zero contribution of non-selected super-pixels would then apply. We also point out that the core algorithm in the majority of the current post-hoc explainers are also domain-agnostic. Therefore, we expect our evaluation to provide a representative view of the fundamental limitations of the explainers. We provide the statistics of our dataset in Table 2. N is the number of instances that we retain with our procedure, |x| is the average length of the reviews, and |S x |, |SR x |, and |N x | are the average numbers of selected tokens, selected tokens that give an absolute difference of prediction of at least 0.1 when eliminated individually, and non-selected tokens, respectively. In parenthesis are the standard deviations. The column %(S Sx = S x) provides the percentage of instances eliminated from the original BeerAdvocate dataset due to a potential handshake. Finally, %(|SR x | = 0) shows the percentage of datapoints (out of the non-handshake ones) further eliminated due to the absence of a selected token with absolute effect of at least 0.1 on the prediction. Proof: We note that if there is a handshake in the instance x, i.e., at least one non-selected token x k ∈ N x is actually influencing the final prediction via an internal encoding of its information into the selected tokens, then the model should have a different prediction when x k is eliminated from the instance, i.e., RCNN(x) = RCNN(x − x k). Equivalently, if RCNN(x − x k) = RCNN(x), then x k could not have been part of a handshake. Thus, if the RCNN gives the same prediction when eliminating all the non-selected tokens, it means that there was no handshake for the instance x, and hence the tokens in N x have indeed zero contribution. Hence, we have that: RCNN(x − N x) = RCNN(x) =⇒ no handshake in x. Since x − N x = S x, Equation 8 rewrites as: RCNN(S x) = RCNN(x) =⇒ no handshake in x. From Equation 2, we further rewrite Equation 9 as: encoder(generator(S x)) = encoder(generator(x)) =⇒ no handshake in x. Since, by definition, generator(x) = S x, we have that: encoder(S Sx) = encoder(S x) =⇒ no handshake in x. Hence, it is sufficient to have S Sx = S x in order to satisfy the right-hand-side condition of Equation 11, which finishes our proof. LIME: nice brown " grolsch " like bottle (good for re-use). pours a dark 1. laces yellow color with a lot of head in the beginning which laces well. 2. yellow very fizzy. smells like fruit, maybe some apple and blueberry. no mouthfeel 3. lot whatsover. besides being wet and a small initial alcohol, i could n't feel 4. dark anything. tastes of fruit and not much alcohol, but i can start to feel a slight 5. of warming as i finish off the bottle. better than most american lagers, but very smooth. i think i would normally drink this too fast. SHAP: nice brown " grolsch " like bottle (good for re-use). pours a dark 1. fizzy yellow color with a lot of head in the beginning which laces well. 2. laces very fizzy. smells like fruit, maybe some apple and blueberry. no mouthfeel 3. head whatsover. besides being wet and a small initial alcohol, i could n't feel 4. nice anything. tastes of fruit and not much alcohol, but i can start to feel a slight 5. yellow warming as i finish off the bottle. better than most american lagers, but very smooth. i think i would normally drink this too fast. L2X: nice brown " grolsch " like bottle (good for re-use). pours a dark 1. laces yellow color with a lot of head in the beginning which laces well. very fizzy. smells like fruit, maybe some apple and blueberry. no mouthfeel 3. lot whatsover. besides being wet and a small initial alcohol, i could n't feel 4. whatsover anything. tastes of fruit and not much alcohol, but i can start to feel a slight 5. nice warming as i finish off the bottle. better than most american lagers, but very smooth. i think i would normally drink this too fast. Figure 5: Explainers' rankings (with the top 5 features on the right-hand side) on an instance from the appearance aspect in our evaluation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

An evaluation framework based on a real-world neural network for post-hoc explanatory methods

We construct a theoretical framework for weakly supervised disentanglement and conducted lots of experiments to back up the theory.