Question: represent the article for finding a single-sentence tldr

Question: Touch interactions with current mobile devices have limited expressiveness. Augmenting devices with additional degrees of freedom can add power to the interaction, and several augmentations have been proposed and tested. However, there is still little known about the effects of learning multiple sets of augmented interactions that are mapped to different applications. To better understand whether multiple command mappings can interfere with one another, or affect transfer and retention, we developed a prototype with three pushbuttons on a smartphone case that can be used to provide augmented input to the system. The buttons can be chorded to provide seven possible shortcuts or transient mode switches. We mapped these buttons to three different sets of actions, and carried out a study to see if multiple mappings affect learning and performance, transfer, and retention. Our show that all of the mappings were quickly learned and there was no reduction in performance with multiple mappings. Transfer to a more realistic task was successful, although with a slight reduction in accuracy. Retention after one week was initially poor, but expert performance was quickly restored. Our work provides new information about the design and use of augmented input in mobile interactions. Mobile touchscreen devices such as smartphones, tablets, and smartwatches are now ubiquitous. The simplicity of touch-based interaction is one of the main reasons for their popularity, but touch interfaces have low expressivenessthey are limited in terms of the number of actions that the user can produce in a single input. As a , touch interactions often involve additional actions to choose modes or to navigate menu hierarchies. These limitations on touch input can be addressed by adding new degrees of freedom to touch devices. For example, both Android and IOS devices have augmentations that allow the user to specify the difference between scrolling and selecting: Android uses a timeout on the initial touch (i.e., a drag starts with either a short press or a long press), and some IOS devices use pressure-sensitive screens that use different pressure levels to specify selection and scrolling. Researchers have also proposed adding a wide variety of new degrees of freedom for touch devices -including multi-touch and bimanual input, external buttons and force sensors, back-of-device touch, sensors for pen state or screen tilt, and pressure sensors. Studies have shown these additional degrees of freedom to be effective at increasing the expressive power of interaction with a mobile device. However, previous research has only looked at these new degrees of freedom in single contexts, and as a we know little about how augmented input will work when it is used in multiple different applications: if an augmented input is mapped to a set of actions that are specific to one application, will there be interference when the same augmentations are mapped to a different set of actions in another application? To find out how multiple mappings for a new degree of freedom affect learning and usage, we carried out a study with a prototype device that provides three buttons on the side of a smartphone case. The buttons can be chorded, giving seven inputs that can be used for discrete commands or transient modes. We developed three different mappings for these chording buttons for three different contexts: shortcuts for a launcher app, colour selections for a drawing app; and modes for a text-editing app. Our study looked at three issues: first, whether learning multiple mappings with the chorded buttons would interfere with learning or accuracy; second, whether people could transfer their learning from training to usage tasks that set the button commands into more complex and realistic activities; and third, whether memory of the multiple mappings would be retained over one week, without any intervening practice. Our evaluation provide new insights into the use of augmented input for mobile devices:  Learning multiple mappings did not reduce performance -people were able to learn all three mappings well, and actually learned the second and third mappings significantly faster than the first;  Multiple mappings did not reduce accuracy -people were as accurate on a memory test with three mappings as they were when learning the individual mappings;  Performance did transfer from training to more realistic usage tasks, although accuracy decreased slightly;  Retention after one week was initially poor (accuracy was half that of the first session), but performance quickly returned to near-expert levels. Our work provides two main contributions. First, we show that chorded input is a successful way to provide a rich input vocabulary that can be used with multiple applications. Second, we provide empirical evidence that mapping augmented input to multiple contexts does not impair performance. Our provide new evidence that augmented input can realistically increase the expressive power of interactions with mobile devices. ), and researchers have proposed new paradigms of interaction (e.g., for eyes-free ubiquitous computing or for post-WIMP devices ) that can incorporate different types of augmentation. Cechanowicz and colleagues also created a framework specifically about augmented interactions; they suggest several ways of adding to an interaction, such as adding states to a discrete degree of freedom, adding an entirely new degree of freedom, or "upgrading" a discrete degree of freedom to use continuous input. Chorded input for text entry has existed for many years (e.g., stenographic machines for court reporters, or Engelbart and English's one-hand keyboard in the NLS system ). Researchers have studied several issues in chorded text input, including performance, learning, and device design. A longitudinal study of training performance with the Twiddler one-handed keyboard showed that users can learn chorded devices and can gain a high level of expertise. The study had 10 participants train for 20 sessions of 20 minutes each; showed that by session eight, chording was faster than the multi-tap technique, and that by session 20, the mean typing speed was 26 words per minute. Five participants who continued the study to 25 hours of training had a mean typing speed of 47 words per minute. Because this high level of performance requires substantial training time, researchers have also looked at ways of reducing training time for novices. For example, studies have investigated the effects of using different types of phrase sets in training, and the effects of feedback. Several chording designs have been demonstrated for text entry on keypad-style mobile phones. The ChordTap system added external buttons to the phone case; to type a letter, the dominant hand selects a number key on the phone (which represents up to four letters) and the non-dominant hand presses the chording keys to select a letter within the group. A study showed that the system was quickly learned, and outperformed multi-tap from the second block of trials. A similar system used three of the keypad buttons to select the letter within the group, allowing chorded input without external buttons. The TiltText prototype used the four directions of the phone's tilt sensor to choose a letter within the group. Other types of chording have also been seen in multi-touch devices, where combinations of fingers are used to indicate different states to the system. Several researchers have looked at multi-touch input for menu selection. For example, finger-count menus use the number of fingers in two different areas of the touch surface to indicate a category (with the left hand) and an item within that menu (with the right hand). Two-handed marking menus also divide the screen into left and right sections, with a stroke on the left side selecting the submenu and a stroke on the right selecting the item. Multitouch marking menus combine these two approaches, using visionbased finger identification to increase the number of possible combinations. Each multi-finger chord indicates which menu is to be displayed, and the subsequent direction in which the touch points are moved indicates the item to be selected. HandMarks is a bimanual technique that uses the left hand on the surface as a reference frame for selecting menu items with the right hand. The FastTap system uses chorded multiple touches to indicate both that the menu mode is active and the selection of an item from a grid menu. Other kinds of chording have also been investigated with touch devices. The BiTouch system was a general-purpose technique that allowed touches from the supporting hand to be used in conjunction with touches from the dominant hand. Olafsdottir and Appert developed a taxonomy of multi-touch gestures (including chords), and Ghomi and colleagues developed a training technique for learning multi-touch chords. Finally, multi-finger input on a phone case was also shown by Wilson and Brewster, who developed a prototype with pressure sensors under each finger holding the phone; input could involve single fingers or combinations of fingers (with pressure level as an added degree of freedom). Researchers have also developed touch devices and techniques that involve other types of additional input, including methods for combining pen input with touch, using vocal input, using the back of the device as well as the front, using tilt state with a directional swipe on the touch surface to create an input vocabulary, or using a phone's accelerometers to enhance touch and create both enhanced motion gestures (e.g., one-handed zooming by combining touch and tilt), and more expressive touch. Enhanced input can also address issues with interface modes, which are often considered to be a cause of errors. Modes can be persistent or "spring loaded" (also called quasimodes ); these are active only when the user maintains a physical action (e.g., holding down a key), and this kinesthetic feedback can help people remember that they are in a different mode. When interfaces involve persistent modes, several means for switching have been proposed. For example, Li and colleagues compared several mode-switch mechanisms for changing from inking to gesturing with a stylus: a pen button, a separate button in the non-dominant hand, a timeout, pen pressure, and the eraser end of the pen. They found that a button held in the other hand was fastest and most preferred, and that the timeout was slow and error prone. Other researchers have explored implicit modes to allow fluid specification of a mode without an explicit switch. For example, Chu and colleagues created pressuresensitive "haptic conviction widgets" that allow either normal or forceful interaction to indicate different levels of confidence. Similarly, some IOS devices use touch pressure to differentiate between actions such as selection and scrolling. Many techniques add new sensing capabilities to create the additional modes -for example, pressure sensors have also been used to enhance mouse input and pen-based widgets; three-state switches were added to a mouse to create pop-through buttons; and height sensing was used to enable different actions in different height layers (e.g., the hover state of a pen, or the space above a digital table ). Other techniques simply use existing sensing that is currently unused in an interaction. For example, OrthoZoom exploits the unused horizontal dimension in a standard scrollbar to add zooming (by moving the pointer left or right while scrolling). Despite the wide range of work that has been carried out in this area, there is relatively little research on issues of interference, transfer, or retention for these augmented interfaces -particularly with multiple mappings. The study below provides initial baseline information for these issuesbut first, we describe the design and construction of the prototype that we used as the basis for our evaluation. In order to test learning, interference, and retention, we developed a prototype system that adds three hardware buttons to a custom-printed phone case and makes the state of those buttons available to applications. We designed and 3D-printed a bumper-style case for an Android Nexus 5 phone, with a compartment mounted on the back to hold the circuit boards from three Flic buttons (Bluetooth LE buttons made by Shortcut Labs). The Flic devices can be configured to perform various predetermined actions when pressed; Shortcut Labs also provides an Android API for using the buttons with custom software. We removed the PCBs containing the Bluetooth circuitry, and soldered new buttons to the PCBs (Figure 1). The new pushbuttons are momentary switches (i.e., they return to the "off" state when released) with 11mm-diameter push surfaces and 5mm travel. We tested several button styles and sizes, in order to find devices that were comfortable to push, that provided tactile feedback about the state of the press, and that were small enough to fit under three fingers. This design allows us to use the Flic Bluetooth events but with buttons that can be mounted closer together. The new buttons do not require any changes to our use of the API. The prototype is held as a normal phone with the left hand, with the index, middle, and ring fingers placed on the pushbuttons (Figure 2). The pushbuttons are stiff enough that these three fingers can also grip the phone without engaging the buttons; the fifth finger of the left hand can be placed comfortably on the phone case, adding stability when performing chorded button combinations. Finally, we note that the button housing on our prototype was larger than would be required by a commercial device; we estimate that the hardware could easily be built into a housing that is only marginally larger than a typical phone case. The prototype worked well in the study sessions described below. No participant complained of fatigue or difficulty pressing the buttons (although we observed a few difficulties matching the timeout period, as described below). The phone case was easy to hold, and the button positions were adequate for the hand sizes of our participants. Pressing the buttons in chords did not appear to cause difficulty for any participant (although with some timing issues, as described later). We wrote a simple wrapper library for Android to attach callback functions to the buttons through the Flic API. Android applications can poll the current combined state of the buttons through this library wrapper. Callback functions attached through the wrapper library are put on a short timer, allowing time for multiple buttons to be depressed before executing the callback. In all the applications we created, we assigned a single callback function to all the buttons; this function checks the state of all buttons and determines the appropriate behavior based on the combined state. Identifying chords represents an interpretation problem for any input system. When only individual buttons can be pressed, software can execute actions as soon as the signal has been received from any button. When chorded input is allowed, however, this method is insufficient, because users do not press all of the buttons of a chord at exactly the same time. Therefore, we implemented a 200ms wait time before processing input after an initial button signal -after this delay, the callback read the state of all buttons, and reported the combined pattern (i.e., a chord or a single press). Once an input is registered, all buttons must return to their "off" states before another input can be produced. With three buttons, the user can specify eight states -but in our applications, we assume that there is a default state that corresponds to having no buttons pressed. This approach prevents the user from having to maintain pressure on the buttons during default operation. We carried out a study of our chording system to investigate our three main research questions:  Interference: does learning additional mappings with the same buttons reduce learning or accuracy?  Transfer: is performance maintained when users move from training to usage tasks that set the button commands into more realistic activities?  Retention: does memory of the command mappings persist over one week (without intervening practice)? To test whether learning multiple mappings interferes with learning rate or accuracy, we created a training application to teach three different mappings to participants: seven application shortcuts (Apps), seven colors (Colors), and seven text-editing commands (Text) (Table 1). Participants learned the mappings one at a time, as this fits the way that users typically become expert with one application through frequent use, then become expert with another. To further test interference, after all mappings were learned we gave participants a memory test to determine whether they could remember individual commands from all of the mappings. This memory test corresponds to scenarios in which the user switches between applications and must remember different mappings at different times. To test whether the mappings learned in the training system would transfer, we asked participants to use two of the mappings in simulated usage tasks. Colors were used in a drawing program where participants were asked to draw shapes in a particular line color, and Text commands were used in a simple editor where participants were asked to manipulate the formatting of lines of text. To test retention, we recruited a subset of participants to carry out the memory test and the usage tasks a second time, one week after the initial session. Participants were not told that they would have to remember the mappings, and did not practice during the intervening week. The first part of the study had participants learn and practice the mappings over ten blocks of trials. The system displayed a target item on the screen, and asked the user to press the appropriate button combination for that item (see Figure 2). The system provided feedback about the user's selection (Figure 2, bottom of screen); when the user correctly selected the target item, the played a short tone, and the system moved on to the next item. Users could consult a dialog that displayed the entire current mapping but had to close the dialog to complete the trial. The system presented each item in the seven-item mapping twice per block (sampling without replacement), and continued for ten blocks. The same system was used for all three mappings, and recorded selection time as well as any incorrect selections (participants continued their attempts until they selected the correct item). We created two applications (Drawing and TextEdit) to test usage of two mappings in larger and more complex activities. Drawing. The Drawing application (Figure 3) is a simple paint program that uses the chord buttons to control line color (see Table 1). The application treated the button input as a set of spring-loaded modes -that is, the drawing color was set based on the current state of the buttons, and was unset when the buttons were released. For example, to draw a red square as shown in Figure 3, users held down the first button with their left hand and drew the square with their right hand; when the button was released, the system returned to its default mode (where touch was interpreted as panning). If the user released the buttons in the middle of a stroke, the line colour changed back to default grey. For each task in the Drawing application, a message on the screen asked the participant to draw a shape in a particular color. Tasks were grouped into blocks of 14, with each color appearing twice per block. A task was judged to be complete when the participant drew at least one line with the correct color (we did not evaluate whether the shape was correct, but participants did not know this). Participants completed three blocks in total. TextEdit. The TextEdit application asked users to select lines of text and apply manipulations such as cutting and pasting the text, setting the style (bold or italic), and increasing or decreasing the font size. Each of these six manipulations was mapped to a button combination. The seventh action for this mapping was used for selection, implemented as a springloaded mode that was combined with a touch action. We mapped selection to the combination of all three buttons since selection had to be carried out frequently -and this combination was easy to remember and execute. For each TextEdit task, the lines on the screen told the user what manipulations to make to the text (see Figure 4). Each task asked the participant to select some text and then perform a manipulation. There were six manipulations in total, and we combined copy and paste into a single task, so there were five tasks. Tasks were repeated twice per block, and there were four blocks. Tasks were judged to be correct when the correct styling was applied; if the wrong formatting was applied, the user had to press an undo button to reset the text to its original form, and perform the task again. The third stage of the study was a memory test that had a similar interface to the learning system described above. The system gave prompts for each of the 21 commands in random order (Apps, Colors, and Text were mixed together, and sampled without replacement). Participants pressed the button combination for each prompt, but no feedback was given about what was selected, or whether their selection was correct or incorrect. Participants were only allowed to answer once per prompt, and after each response the system moved to the next item. To determine participants' retention of the mappings, after the study was over we recruited 8 of the 15 participants to return to the lab after one week to carry out the memory test and the usage tasks again (two blocks of each of the drawing and text tasks). Participants were not told during the first study that they would be asked to remember the mappings beyond the study; participants for the one-week follow-up were recruited after the initial data collection was complete. The usage and memory tests operated as described above. After completing an informed consent form and a demographics questionnaire, participants were shown the system and introduced to the use of the external buttons. Participants were randomly assigned to a mapping-order condition (counterbalanced using a Latin square), and then started the training tasks for their first mapping. Participants were told that both time and accuracy would be recorded but were encouraged to use their memory of the chords even if they were not completely sure. After the Color and Text mappings, participants also completed the usage tasks as described above (there was no usage task for the Apps mapping). After completing the learning and tasks with each mapping, participants filled out an effort questionnaire based on the NASA-TLX survey. After all mappings, participants completed the memory test. For the retention test, participants filled out a second consent form, then completed the memory test with no assistance or reminder of the mappings. They then carried out two blocks of each of the usage tasks (the Drawing and TextEdit apps had the same order as in the first study). Fifteen participants were recruited from the local university community (8 women, 7 men, mean age 28.6). All participants were experienced with mobile devices (more than 30min/day average use). All but one of the participants was right-handed, and the one left-handed participant stated that they were used to operating mobile devices in a righthanded fashion. The study used the chording prototype described above. Sessions were carried out with participants seated at a desk, holding the phone (and operating the chording buttons) with their left hands. The system recorded all performance data; questionnaire responses were entered on a separate PC. The main study used two 3x10 repeated-measures designs. The first looked at differences across mappings, and used factors Mapping (Apps, Colors, Text) and Block. The second looked at interference by analyzing differences by the position of the mapping in the overall sequence, and used factors Position (first, second, third) and Block. For the memory tests, we used a 21x3x7 design with several planned comparisons; factors were Item (the 21 items shown in Table 1), Pattern (the 7 button patterns shown in column 2 of Table 1), and Mapping (Apps, Colors, Text). Dependent variables were selection time, accuracy (the proportion of trials where the correct item was chosen on the first try), and errors. No outliers were removed from the data. In the following analyses, significant ANOVA report partial etasquared as a measure of effect size (where .01 can be considered small, .06 medium, and > .14 large ). We organize the below around the main issues under investigation: training performance when learning three different mappings, interference effects, transfer performance, and retention after one week..09), with no interaction (F18,252=0.71, p=0.78). Overall error rates (i.e., the total number of selections per trial, since participants continued to make selections until they got the correct answer) for all mappings were low: 0.25 errors / selection for Apps, 0.26 for Colors, and 0.24 for Text. During the sessions we identified a hardware-based source of error that reduced accuracy. The 200ms timeout period in some cases caused errors when people held the buttons for the wrong period of time, when the Bluetooth buttons did not transmit a signal fast enough, or when people formed a chord in stages. This issue contributes to the accuracy rates shown above: our observations indicate that people had the button combinations correctly memorized but had occasional problem in producing the combination with the prototype. We believe that this difficulty can be fixed by adjusting our timeout values and by using an embedded microprocessor to read the button states (thus avoiding Bluetooth delay). Perceived Effort. Responses to the NASA-TLX effort questionnaire are shown in Figure 7; overall, people felt that all of the mappings required relatively low effort. Friedman rank sum tests showed only one difference between mappings -people saw themselves as being less successful with the Apps mapping (χ 2 =7, p=0.030). In the end-of-session questionnaire, 12 participants stated that Colors were easiest to remember (e.g., one person stated "colours were easier to remember" and another said that "memorizing the colours felt the easiest"). To determine whether learning a second and third mapping would be hindered because of the already-memorized mappings, we analysed the performance data based on whether the mapping was the first, second, or third to be learned. Figure 9 shows selection time over ten blocks for the first, second, and third mappings (the specific mapping in each position was counterbalanced). Selection time. A 3x10 RM-ANOVA looked for effects of position in the sequence on selection time. We did find a significant main effect of Position (F2,28=19.68, p<0.0001, η 2 =0.22), but as shown in Figure 8, the second and third mappings were actually faster than the first mapping. Both subsequent mappings were faster than the first; follow-up ttests with Bonferroni correction show that these differences are significant, p<0.01). The difference was more obvious in the early blocks (indicated by a significant interaction between Position and Block, F18,252=4.63, p<0.0001, η 2 =0.14). These findings suggest that adding new mappings for the same buttons does not impair learning or performance for subsequent mappings. Accuracy. We carried out a similar 3x10 RM-ANOVA to look for effects on accuracy (Figure 9). As with selection time, performance was worse with the first mapping (accuracy 0.8) than the second and third mappings (0.85 and 0.86). ANOVA showed a main effect of Position on accuracy (F2,28=7.18, p=0.003, η 2 =0.072), but with no Position x Block interaction (F18,252=1.20, p=0.051). The third stage of the study was the memory test, in which participants selected each of the 21 commands from all three mappings, in random order. Participants answered once per item with no feedback. The overall accuracy was 0.87 (0.86 for Apps, 0.86 for Colors, and 0.89 for Text); see Figure 11. Note that this accuracy is higher that seen with the individual mappings during the training sessions. To determine whether there were differences in accuracy for individual items, mappings, or button patterns, we carried out a 21x3x7 (Item x Mapping x Pattern) RM-ANOVA. We found no significant effects of any of these factors (for Item, F20,240=1.55, p=0.067; for Mapping: F2,24=0.43, p=0.65; for Pattern, F6,12=0.0004, p=0.99), and no interactions. Figure 10 (note that the text task had four blocks, and the drawing task had three blocks). Accuracy in the usage tasks ranged from 0.7 to 0.8 across the trial blocks -slightly lower than the 0.8-0.9 accuracy seen in the training stage of the study. It is possible that the additional mental requirements of the task (e.g., determining what to do, working with text, drawing lines) disrupted people's memory of the mappings -but the overall difference was small. The one-week followup asked eight participants to carry out the memory test and two blocks of each of the usage tasks, to determine whether participants' memory of the mappings had persisted without any intervening practice (or even any knowledge that they would be re-tested). Overall, the followup showed that accuracy decayed substantially over one week -but that participants quickly returned to their previous level of expertise once they started the usage tasks. In the memory test, overall accuracy dropped to 0.49 (0.43 for Apps, 0.50 for Colors, and 0.54 for Text), with some individual items as low as 10% accuracy. Only two items maintained accuracy above 0.85 -"Red" and "Copy". The two usage tasks (Drawing and Text editing) were carried out after the memory test, and in these tasks, participant accuracy recovered considerably. In the first task (immediately after the memory test), participants had an overall 0.60 accuracy in selection; and by the second block, performance rose to accuracy levels similar to the first study (for Drawing, 0.82; for Text, 0.70). This follow-up study is limited -it did not compare retention when learning only one mapping, so it is impossible to determine whether the decay arose because of the number of overloaded mappings learned in the first study. However, the study shows that retention is an important issue for designers of chorded memory-based techniques. With only a short training period (less than one hour for all three mappings) appears to be insufficient to ensure retention after one week with no intervening practice; however, in an ecological context users would likely use the chords more regularly. In addition, participants' memory of the mappings was restored after only a few minutes of use. Our study provides several main findings:  The training phase showed that people were able to learn all three mappings quickly (performance followed a power law), and were able to achieve 90% accuracy within a few training blocks;  Overloading the buttons with three different mappings did not appear to cause any problems for participantsthe second and third mappings were learned faster than the first, and there was no difference in performance across the position of the learned mappings;  People were able to successfully transfer their expertise from the training system to the usage tasks -although performance dropped by a small amount;  Performance in the memory test, which mixed all three mappings together, was very strong, with many of the items remembered at near 100% accuracy;  Retention over one week without any intervening practice was initially poor (about half the accuracy of the first memory test), but recovered quickly in the usage tasks to near the levels seen in the first sessions. In the following paragraphs we discuss the reasons for our , and comment on how our findings can be generalized and used in the design of richer touch-based interactions. People's overall success in learning to map twenty-one total items to different button combinations is not particularly surprising -evidence from other domains such as chording keyboards suggests that with practice, humans can be very successful in this type of task. It is more interesting, however, that these 21 items were grouped into three overloaded sets that used the same button combinations -and we did not see any evidence of interference between the mappings. One reason for people's success in learning with multiple button mappings may be that the contexts of the three mappings were quite different, and there were few conceptual overlaps in the semantics of the different groups of items (e.g., colors and application shortcuts are quite different in the ways that they are used). Even if this is true, however, there are likely many opportunities in mobile device use where this type of clean separation of semantics occurs -suggesting that overloading can be used to substantially increase the expressive power of limited input. People were also reasonably successful in using the learned commands in two usage tasks. This success shows that moving to more realistic tasks does not substantially disrupt memories built up during a training exercise -although it is likely that the added complexity of the tasks caused the small observed reduction in accuracy compared to training. The overall difference between the training and usage environments was relatively small, however; more work is needed to examine transfer effects to real-world use. Finally, the decay in memory of the mappings over one week may simply be an effect of the human memory system -our training period was short example, Ebbinghaus's early studies on "forgetting curves" show approximately similar decay to what we observed. It is likely that in real-world settings, the frequency of users' mobile phone use would have provided intervening practice that would have maintained users' memory -but this must be studied in greater detail in future. Limitations and opportunities for future work The main limitations of our work are in the breadth and realism of our evaluations, and in the physical design of the prototype. First, although our work takes important steps towards ecological validity for augmented input, our study was still a controlled experiment. We designed the study to focus on real-world issues of interference, transfer, and retention but the realism of our tasks was relatively low. Therefore, a critical area for further work is in testing our system with real tasks in real-world settings. The Flic software allows us to map button inputs to actions in real Android applications, so we plan to have people use the next version of the system over a longer time period and with their own actual applications. Second, it is clear that additional engineering work can be done to improve both the ergonomics and the performance of the prototype. The potential errors introduced by our 200ms timeout are a problem that can likely be solved, but the timeout caused other problems as well -once participants were expert with the commands, some of them felt that holding the combination until the application registered the command slowed them down. Adjusting the timeout and ensuring that the system does not introduce additional errors is an important area for our future work. We also plan to experiment with different invocation mechanisms (e.g., selection on button release) and with the effects of providing feedback as the chord is being produced. An additional opportunity for future work that was identified by participants during the study is the potential use of external chorded buttons as an eyes-free input mechanism. The button interface allows people to change input modes without shifting their visual attention from the current site of work, and also allows changing tools without needing to move the finger doing the drawing (and without occluding the workspace with menus or toolbars). Expressiveness is limited in mobile touch interfaces. Many researchers have devised new ways of augmenting these interactions, but there is still little understanding of issues of interference, transfer, and retention for augmented touch interactions, particularly those that use multiple mappings for different usage contexts. To provide information about these issues, we developed an augmented phone case with three pushbuttons that can be chorded to provide seven input states. The external buttons can provide quick access to command shortcuts and transient modes, increasing the expressive power of interaction. We carried out a four-part study with the system, and found that people can successfully learn multiple mappings of chorded commands, and can maintain their expertise in more-complex usage tasks. Retention was found to be an important issue -accuracy dropped over one week, but was quickly restored after a short period of use. Our work provides new knowledge about augmented interactions for touch devices, and shows that adding simple input mechanisms like chorded buttons are a promising way to augment mobile interactions.

represent the tldr for finding the corresponding scientific article

Describes a study investigating interference, transfer, and retention of multiple mappings with the same set of chorded buttons

represent this text to find a much much longer article about it with say hundreds-thousands of words:

We automatically extract fingering information from videos of piano performances, to be used in automatic fingering prediction models.


Question: represent the article for finding a single-sentence tldr

Question: Neural networks are widely used in Natural Language Processing, yet despite their empirical successes, their behaviour is brittle: they are both over-sensitive to small input changes, and under-sensitive to deletions of large fractions of input text. This paper aims to tackle under-sensitivity in the context of natural language inference by ensuring that models do not become more confident in their predictions as arbitrary subsets of words from the input text are deleted. We develop a novel technique for formal verification of this specification for models based on the popular decomposable attention mechanism by employing the efficient yet effective interval bound propagation (IBP) approach. Using this method we can efficiently prove, given a model, whether a particular sample is free from the under-sensitivity problem. We compare different training methods to address under-sensitivity, and compare metrics to measure it. In our experiments on the SNLI and MNLI datasets, we observe that IBP training leads to a significantly improved verified accuracy. On the SNLI test set, we can verify 18.4% of samples, a substantial improvement over only 2.8% using standard training. Natural language processing (NLP) widely relies on neural networks, a model class known to be vulnerable to adversarial input perturbations . Adversarial samples typically expose over-sensitivity to semantically invariant text transformations , e.g. character flips or paraphrases (b; . exposed another type of problematic behavior: deleting large parts of input text can cause a model's confidence to increase; Figure 1 shows an example. That is, reduced sets of input words can suffice to trigger more confident predictions. Such under-sensitivity is problematic: neural models can 'solve' NLP tasks without task-relevant textual comprehension skills, but instead fit spurious cues in the data that suffice to form correct predictions. Models might then achieve strong nominal test accuracy on data of the same (biased) distribution as the training set, by exploiting predictive shortcuts that are not representative of the given NLP task at hand. Consequently, they fail drastically when evaluated on samples without these spurious cues (; ; ;). A major issue with identifying reduced inputs is the combinatorially large space of arbitrary text deletions; this can only be searched exhaustively for short sequences. Prior work has considered heuristics like beam search or bandits (a), but these are generally not guaranteed to find the worst-case reductions. In this work, we address the under-sensitivity issue by designing and formally verifying the undersensitivity specification that a model should not become more confident as arbitrary subsets of input words are deleted. 1 Under-sensitivity behaviour is not reflected in nominal accuracy, but one can instead use this specification to measure and evaluate the extent with which samples exhibit undersensitivity. Instead of better, yet still imperfect search heuristics, we describe how interval bound propagation (IBP) -a formal model verification methodcan be used to efficiently cover the full reduction space, and verify the under-sensitivity specification. IBP can be applied at test time to arbitrary model inputs to verify whether or not they are undersensitive; but it can also be used to derive a new auxiliary training objective that leads to models verifiably adhering to this specification, and which we find generalizes to held-out test data. While under-sensitivity has been demonstrated for several NLP tasks , we chose to study the use case of natural language inference (NLI) in particular as a representative task: sequences are comparatively short, datasets large, and the label complexity is small. We investigate the verification of the popular decomposable attention model (DAM) 2 in detail. This architecture covers many of the neural layer types of contemporary models, and we focus on a detailed description for how IBP can be leveraged to efficiently verify its behaviour. We then experimentally compare various training methods addressing under-sensitivity: i) standard training ii) data augmentation iii) adversarial training iv) IBP-verified training and v) entropy regularization, and evaluate their effectiveness against nominal (test) accuracy, adversarial accuracy, IBP-verified accuracy and a verification oracle. To summarise, the main contributions of this paper are Formalization of the problem of verifying an under-sensitivity specification, Verification of the Decomposable Attention Model using Interval Bound Propagation, and Empirical analysis of the efficacy of (i) different evaluation methods for verifying robustness; and (ii) different training methods for developing verifiably robust models. Natural Language Inference. Natural Language Inference is the task of predicting whether a natural language premise entails a natural language hypothesis. The availibility of large-scale datasets has spurred a profusion of neural architecture development for this task, e.g. (Rocktäschel et al., 2016; ;), among many others. Adversarial Vulnerability in NLP. There is a growing body of research into NLP adversarial examples, each using a slightly different choice of semantically invariant text transformations, or a task-specific attack. A first class of attack considers word-and character-level perturbation attacks while another type of attack exploits back-translation systems to either mine rules (b) or train syntactically controlled paraphrasing models. use syntactic and lexical transformations, whereas investigate synthetic and natural noise in Machine Translation. and introduce task-specific adversarial attacks for Reading Comprehension/QA, for for Fact Checking. In NLI in particular, penalize adversarially chosen logical inconsistencies in NLI predictions, use -knowledge guided adversaries, and showing that models can become more confident as large fractions of input text are deleted, whereas address under-sensitivity in a dialogue setting. demonstrated a link between excessive prediction invariance and model vulnerability in computer vision. Formal Verification. Formal verification provides a provable guarantee that models are consistent with a formally defined specification (a mathematical relationship between the inputs and outputs of the model). Examples of specifications include robustness to bounded adversarial perturbations, monotonicity of the output wrt a subset of the inputs, and consistency with physical laws . Literature can be categorised into complete methods that use Mixed-Integer Programming (MIP) or Satisfiability Modulo Theory (SMT) , and incomplete methods that solve a convex relaxation of the verification problem (; ; ; b). Complete methods perform exhaustive enumeration to find a counter-example to the specification or rule out the existence of counter-examples (hence proving that the specification is true). Hence, complete methods are expensive and difficult to scale. Incomplete methods are conservative (i.e., they cannot always prove that a specification is true even when it is), but are more scalable and can be used inside the training loop for training models to be consistent and verifiable (a; ; a; ; b). address the issue of verification in NLP, most of the recent work has focused on ∞ norm-bounded perturbations for image classification. This paper complements work on incomplete verification methods by extending IBP to NLI where inputs are inherently discrete (to the contrary of images which are continuous). In the NLP context in particular, and have very recently verified CNN, and LSTM models with specifications against over-sensitivity adversaries under synonym replacement. study verification of output length specifications in machine translation models, showing that the outputs of machine translation and image captioning systems can be provably bounded when the inputs are perturbed within a given set. In contrast, this work examines under-sensitivity behaviour: excessive model prediction invariance under arbitrary word combination deletions. We highlight that the verification of neural networks is an extremely challenging task, and that scaling complete and incomplete methods to large models is an open problem. Neural networks are expressive models that can fit large datasets and achieve strong nominal test accuracy. At the same time however, they can fit data in a way that violates our idea of how they should fit it, from an input attribution perspective. Figure 2 visualises the extent of the problem in NLI: it is, for example, for 20% of the SNLI test set possible to delete 78% or more of premise words while the prediction confidence increases or remains the same. We will next formally describe a specification that checks a model's under-sensitivity, i.e. whether any such reduction exists. The specification addresses model output probabilities when parts of the input text are deleted. To this end, we first introduce the notion of a perturbation space X in (x nom) of an original nominal input x nom. This space contains all possible reductions, i.e. inputs where arbitrarily chosen tokens of the original nominal input x nom are deleted. Note that this space grows exponentially in the length of the input. We would like to verify whether or not there exists any reduced input x − with higher probability for the (nominal) prediction than x nom has. More formally, this can be stated as a specification: whereŷ is the (nominal) model prediction. Determining precisely how prediction probabilities should change when input words are deleted is contentious and prone to inconsistencies: removing stop words for example may lead to little relevant change, while crucial information carrying words (e.g., 'not') can significantly alter the meaning of the sentence in a task. It is important to be cautious and not too restrictive in the specification design, and certain that whatever is specified is desirable. A specification to at least not increase prediction probabilities under arbitrary input deletion is a conservative choice. 3 Other specifications are worth consideration, such as monotonically decreasing certainty as more input is deleted. We will however see that even our very conservative choice of an under-sensitivity specification is hard to positively verify for most inputs in the DAM model. There are different approaches to establish if the Specification is satisfied. With unlimited computational capacity, the property could exhaustively be evaluated for all x − ∈ X in (x nom). Statistically sampling from the reduction space can give an indication of under-sensitivity, but has a very limited coverage rate. Search heuristics can try to identify violations (and be used for 'adversarial' training), but there is no guarantee that a stronger search procedure cannot find more or worse violations. IBP verification on the other hand offers a formal guarantee across the whole space by establishing outer bounds for X in (x nom) and ing bounds on output probabilities. We next give a brief introduction to the Decomposable Attention Model (DAM) which we will later verify. The DAM architecture comprises commonly used neural NLP components, such as word embeddings, attention, and feed-forward networks. Subsequently we introduce Interval Bound Propagation and then bring these together to verify the behaviour of the DAM, i.e. efficiently assert whether an input satisfies Specification. Decomposable Attention. The NLI task takes two word sequences as input -a premise and a hypothesis -and outputs a discrete entailment label prediction in {entailment, neutral, contradiction}. The DAM architecture assumes the input word sequences to be embedded (e.g. as d-dimensional word vectors), i.e. operates on two sequences of input vectors 4:.] denotes concatenation, and I and J are sequence lengths. Word vectors are individually transformed with a vector-valued function F , and pairs thereof are then compared: Note that we follow the notation of , and e ij is not related to a basis vector. In the general model formulation F can be a linear transformation or MLP; this does not affect the derivations made here. Adopting matrix notation across position pairs (i, j), Equation can instead be rewritten in matrix form as E = F (A) F (B) ∈ R I×J, which is used to compute two attention masks -one over each sequence -by normalising across i or across j: These two attention masks serve as coefficients in a convex combination over the original word vectors, aggregating each of the two sequences: That is, A and B hold attention-aggregated word vectors from A and B at positions j = 1,..., J and i = 1,..., I, respectively. These are joined with the original word representations, mixed using a position-wise feed-forward network G: R 2d → R d and finally summed into a single vector representation for each sequence: As a last step, a logit vector with entries for each class is computed as H ([v 1, v 2] ), where H: R 2d → R C is again a feed-forward network, and C the number of output classes. Interval Bound Propagation. IBP is an incomplete but efficient verification method that can be used to verify input-output relationships. It tracks how a part of the input space (in our case: the perturbation space X in (x nom)) propagates forward through the network. IBP starts with an axisaligned bounding box surrounding X in (x nom), and uses interval arithmetic to obtain an axis-aligned bounding box for the output set. Formally, let us assume that the neural network is defined by a sequence of transformations h k for each of its K layers. That is, for z 0 ∈ X in (x nom) The output z K ∈ R C has C logits corresponding to C classes. IBP bounds the activation z k of each layer by an axis-aligned bounding box (i.e., z k ≤ z k ≤ z k) using interval arithmetic. We have for each coordinate z k,i of z k: z k,i = min where e i is the standard i th basis vector. Finally, at the last layer, an upper bound on the worst-case violation of the specification can be evaluated quickly from the logit lower and upper bounds z K and z K respectively, as the bounds translate directly into bounds for the softmax probabilities. IBP can be performed in parallel while running the nominal forward pass. However in general the output bounds are loose, which is exacerbated with increasing network depth. Consequently IBP over-approximates the true extent of the image of X in (x nom) in output space, and can in false negatives. It is thus in practice important to keep the bounds as tight as possible. IBP can be used at test time for verification, but also for training, minimising loss terms derived from the logit bounds. IBP has been used on MLPs and convolutional networks with monotonic activations (; . One technical contribution of this work is to apply it to a model with an attention component (Section 5). To address under-sensitivity, we aim to verify Specification for the DAM model. If the upper probability bound z K of the entire perturbation space X in (x nom) is smaller than the probability P (ŷ|x nom) for the predicted class, then the specification is verified. That is ∀z 0 ∈ X in (x nom): Using this inequality, we can assert whether Specification is verifiably satisfied for any given x nom, i.e. whether or not there exist any reduced samples with higher probability than x nom. Overview. We will first describe the model behaviour when removing a single word at fixed position, and then extend this to deleting single words at any position, and finally generalize this to arbitrary multi-token deletions. One key difference to IBP bounds of other architectural components, such as CNNs or feed-forward layers, is the need for bounds on the attention normalisation, which has to take into account per-token upper and lower bounds. We will exploit the fact that each vector of B is a convex combination of the J vectors that constitute B (and similarly for A). Hence, component-wise bounds on B can be obtained efficiently by maximising over those J vectors. A and B are then inputs to a regular feed-forward network (G followed by H), for which IBP can be used. Deleting Single Word: Particular Position. We first describe how model variables behave when an individual token at a fixed position r is removed from one of the sequences. Without loss of generality, we delete words from the second sequence, noting that the model has a symmetric architecture and that the same can be derived for the other input sequence. We denote all ing quantities with a bar (as inB). That is, when removing a single token at position r: whereasĀ = A. Since F is applied per-position in the sequence, the effect of word deletion remains isolated at this point; the matrix productĒ = F (Ā) F (B) ∈ R I×(J−1) has identical entries as before, but the r-th column disappears. (A) has identical entries compared to P (A), but the r-th column removed. That is, for i = 1,..., I and j = 1,..., J s.t. j = r: The attention maskP (B) on the other hand has renormalized entries. The values retain their relative order, yet the entries are larger because the r-th normalisation summand is removed. For j = r: Hence we can computeP (B) ij in closed form as To summarise the above: attention weights P (B) ij remain largely unchanged when deleting token r, but are rescaled to take into account missing normalisation mass. In the next step, the model computes convex combinationsĀ andB. Concretely,Ā =Ā ·P (A) ∈ R d×(J−1) has unchanged elements compared to before (as A remains unchanged), but the r-th column is removed. ForB =B · (P (B) ) ∈ R d× I the dimensionality remains unchanged, butB has fewer elements and (P (B) ) is renormalised accordingly. Note that all this can still be computed in closed form using Equation, i.e. without need for IBP thus far, and these quantities can further be fed through the remaining network layers G and H to obtain probabilities. Deleting Single Word: Arbitrary Position. We have reached the point whereĀ,B,Ā andB are derived in closed form, for fixed position r. These can be computed exactly without approximation, for deleted words at any position r in the sequence. Extending this to arbitrary single-word deletions, we take the elementwise minimum/ maximum across all possible single word deletions, e.g. which establishes upper and lower bounds for each element, and analogously for the other matrices. In the DAM architecture, these matrices are next fed into dense feed-forward layers G (Equations and) and H, each with two layers. 5 We use IBP to propagate bounds through these layers, feeding in bounds onĀ,B,Ā andB as described above. As a , after propagating these bounds through G and H, we obtain bounds on output logits (and consequently on probabilities) for deletions of a single token at any position. One further simplification is possible: we computev 2 directly from v 2 by subtracting the r-th summand for fixed r (see Equation). Generalising this to arbitrary positions r, we can bound the subtracted vector with max r=1,...,J {G([Ā r ;b r])} and min r=1,...,J {G([Ā r ;b r])}, and thus directly obtain bounds forv 2. Deleting Several Words. We have described the behaviour of intermediate representations (and bounds for them) under deletions of arbitrary individual words; the case of removing several words is similar. The values of remaining individual word vectors a i and b j naturally remain unchanged. The previously established bounds for single word deletions can be partly re-used to establish bounds for arbitrary multi-word deletions, see appendix A for more detail. The ing bounds forv 1 and v 2 are then input to a regular feed-forward network, for which IBP can be used. We now evaluate to which extent the DAM model verifiably satisfies the Specification against under-sensitivity, and we will furthermore compare different training approaches. Experiments are conducted on two large-scale NLI datasets: SNLI and multiNLI , henceforth MNLI. addressed deletions of hypothesis words in SNLI, we establish the phenomenon also for MNLI, and for premise reductions. In our experiments we use premise reductions, noting that under-sensitivity is also present for hypotheses (see Fig. 2). For SNLI we use standard dataset splits, tuning hyperparameters on the development set and reporting for the test set. For MNLI we split off 2000 samples from the development set for validation purposes and use the remaining samples as test set. We use the same types of feed-forward components, layer size, dropout, and word embedding hyperparameters described by. We evaluate with respect to the following metrics: 1. Accuracy: Standard test accuracy. 2. Verified Accuracy: This metric measures whether both i) the prediction is correct ii) it can be verified that no reduction with higher probability exists, using IBP verification. 3. Beam Search Heuristic: This metric uses beam search to find specification violations in the perturbation space, following the protocol of. Search begins from the full sequence, gradually deleting words while keeping a beam of width 10. This metric then measures whether both i) the search heuristic found no counterexample, and ii) the prediction is correct. Note that this heuristic does not cover the full perturbation space, i.e. does not suffice to rule out counterexamples to the specification. This metric provides an upper bound for verified accuracy. Training Methods We will compare the following training methods: 1. Standard Training: This provides a baseline for under-sensitivity behaviour under standard log-likelihood training. 2. Data Augmentation: A first and comparatively simple way to address under-sensitivity is by adding training samples with random word subsets deleted, and penalizing the model with a loss proportional to the specification violation. 3. Adversarial Training: Here we use a more systematic approach than random word deletions: we search within the perturbation space for inputs with large differences between nominal prediction probability and reduced probability, i.e. the strongest specification violations. We compare both i) random adversarial search that samples 512 randomly reduced Table 1: Experimental : accuracy vs. verified accuracy using IBP, for different training methods. All models tuned for verified accuracy, numbers in %. perturbations and picks the strongest violation, and ii) beam search with width 10, following the protocol of. Both for data augmentation and adversarial training, altered samples are recomputed throughout training. observed that entropy regularization on prediction probabilities can partially mitigate the severity of under-sensitivity. Here we use IBP verification as described in Section 5, which provides upper bounds on the prediction probability of arbitrarily reduced inputs (Eq.). We penalize the model with an auxiliary hinge loss on the difference between the upper probability bound for the gold label y and the nominal probability P (y|x nom). Note that the upper bound serves as a proxy for the adversarial objective, as it over-approximates the probabilities of arbitrary reduced samples, covering the full reduction space comprehensively. Training Details The training methods described above make use of an additive contribution to the training loss besides standard log-likelihood. We tune the scale of the respective contribution in [0.01, 0.1, 1.0, 10.0, 100.0]. All experiments used a learning rate of 0.001, Adam optimizer, and batch size 128. We perform early stopping with respect to verified accuracy, for a maximum of 3M training steps. For verified training, we found it useful to continuously phase in the volume of the perturbation space to its maximum size, similar to. Concretely, we compute the per-dimension center of upper and lower bound, and start linearly increasing its volume until it reaches the full perturbation space volume. Similarly we phase in the perturbation radius, i.e. the maximum number of words deleted from 1 to the maximum sequence length of 48. We tune phasein intervals in training steps. We also experimented with over-inflating the perturbation volume to larger than its real size at training time, as well as randomly sampling a maximum perturbation radius during training, neither of which improved verifiability . Evaluating the Effectiveness of IBP for Verification. Tables 1a and 1b show the main . For both datasets, a non-negligible portion of data points can be verified using IBP. The gap between (standard) accuracy however is striking: only a small fraction of correctly predicted inputs is actually verifiably not under-sensitive. Note that IBP accuracy is naturally bounded above by the beam search heuristic, which does however not cover the full reduction space, and overestimates verification rates. IBP verification becomes particularly effective when adding the IBP-verifiability objective during training, verifying 18.36% and 17.44% of samples on SNLI and MNLI. Verifiability does however come at a cost: test accuracy is generally decreased when tuning for verifiability, compared to. This highlights a shortcoming of test accuracy as a metric: it does not reflect the under-sensitivity problem. Once under-sensitivity is taken into account by dedicated training objectives, or by tuning for verification rates, nominal accuracy suffers. Computational Efficiency of IBP Verification. Table 2 gives a breakdown of the computational cost incurred for verification, both empirically, and the theoretical worst-case number of forward passes required per sample. IBP verification comes with small computational overhead compared to a standard forward pass, which is incurred by propagating upper and lower interval bounds through the network once. A full oracle is computationally infeasible, instead we used an exhaustive search oracle, but only up to a maximum budget of 200K forward passes per sample. Even when stopping as soon as a single reduced sample is found that violates the specification, the incurred time is orders of magnitude larger than verification via IBP. Comparing Training Methods. We next discuss the differences between training methods, and how they reflect in verified model behaviour. In absolute terms, standard training does not adhere to the under-sensitivity specification well, neither on SNLI nor MNLI. Data augmentation and random adversarial training lead to slightly different on the two datasets, albeit without major improvements. These methods have a strong random component in their choice of deletions, and this tends to lead to lower verification rates on MNLI, where premises are on average 6.2 tokens longer, and the reduction space is correspondingly larger. Beam search adversarial training leads to improved verification rates on SNLI, yet not for MNLI, and it is noteworthy that when also trained with beam search adversarial samples, beam search evaluation improves substantially. Entropy regularization improves verified accuracy over standard training; this is in line with previous observations that it mitigates under-sensitivity behaviour, made by. Finally, the dedicated IBP-Training objective substantially raises verification rates compared to all other approaches. In an ablation (Table 3) we evaluate performance on short sequences (up to 12 tokens) in the SNLI test set: here an exhaustive search over all possible reductions is feasible. Still the absolute verification rates are low in absolute terms, but we observe that shorter sequences are comparatively easier to verify, and that the incomplete IBP verification can approach the verification levels of the complete oracle (see rows 1,2,5, and 6). For adversarial training (rows 3 and 4), however, oracle verification rates are much closer to the Beam Search Heuristic. This suggests that i) for short sequences the smaller perturbation space can be covered better by beam search, and ii) adversarial training can lead to high verifiability on short sequences, but it fits a model in a way that in loose IBP bounds. Verification of a specification offers a stronger form of robustness than robustness to adversarial samples. Adversarial accuracy, as e.g. derived from beam search, might conceptually be easier to compute, yet has no guarantees to find all or the strongest violations. In fact, evaluating against weak adversaries under-estimates the extent of a problem and may lead to a false sense of confidence. IBP verification can provide guarantees on the nonexistence of reduced inputs, but it is incomplete and can have false negatives. Observations of comparatively low verification or adversarial accuracy rates-as in this workare not new, and have been found to be a general problem of datasets with high sample complexity . We emphasise that under-sensitivity is a very challenging problem to address; even the relatively conservative specification of non-increasing probability under deletion cannot be fulfilled for the majority of test samples under the baselines tested. We see the verification of the attention-based DAM model as a stepping stone towards the verification of larger and more performant attention-based architectures, such as BERT. Following the derivations here, token deletion bounds could similarly be propagated through BERT's self-attention layer. Towards this end, however, we see two main hurdles: i) BERT's network depth, ing in gradually looser IBP bounds ii) BERT's word piece tokenisation, which requires special consideration in conjunction with token-level perturbations. We have investigated under-sensitivity to input text deletions in NLI and recast the problem as one of formally verifying a specification on model behaviour. We have described how Interval Bound Propagation can be used in order to verify the popular Decomposable Attention Model, and have then compared several training methods in their ability to address and be verified against undersensitivity. We observed that only a relatively small fraction of data points can be positively verified, but that IBP-training in particular is capable of improving verified accuracy.

represent the short summary for finding the corresponding scientific article (long)

Formal verification of a specification on a model's prediction undersensitivity using Interval Bound Propagation

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Training with larger number of parameters while keeping fast iterations is an increasingly adopted strategy and trend for developing better performing Deep Neural Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We show that, unlike previous 8-bit precision training methods, the proposed method works out of the box for representative models: ResNet50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters or keeping certain layers in single precision. We introduce two learnable statistics of the DNN tensors - shifted and squeezed factors that are used to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in information due to quantization. Deep neural networks have achieved state-of-the-art performance on a wide variety of computer vision, audio, and natural language processing (NLP) tasks. This has ed in an explosion of interest around techniques to reduce the memory footprint and energy consumption of neural network training and inference . Although there are a number of methods to address some of these issues for inference, the most effective method for training is using reduced precision numerical formats. While 32-bit floating point (FP32) is the most common data format for neural network training, recent hardware have leveraged techniques that allow for training with 16-bit data formats (Köster et al., 2017;). However, 8-bit precision training remains an open challenge . Current FP8 training methodologies (; require either specialized chunk-based accumulation, stochastic rounding techniques, loss scaling or maintaining some layers of the network in higher precision. Tuning these knobs is non-intuitive and requires significant experimentation for each individual network. Accelerating the adoption of 8-bit data in training DNNs requires a hardware-friendly and out-ofthe-box implementation of FP8. Due to the reduced number of mantissa bits, 8-bit multipliers are smaller and consume less power compared to higher bit representations. In this work we describe a novel 8-bit floating point (FP8) format -shifted and squeezed FP8 (S2FP8) -which has the following advantages compared to previously proposed 8-bit training methodologies: • S2FP8 eliminates the need for loss scaling, which requires significant tuning of the loss scale values and schedule for individual topologies • Leveraged by the forward and backward passes of model training, S2FP8 is effective in adjusting the range of gradients and also of activations and weights • S2FP8 does not require keeping the first and last layer in FP32 precision, which is needed for other approaches, however maintains the master weights and accumulations inside the matrix multipliers in FP32 We demonstrate across image classification, translation, and recommendation models that S2FP8 outperforms previous 8-bit approaches, and reaches the accuracy of FP32 models without any additional hyperparameter tuning. The success of 32-bit floating point data type in training deep neural networks has increased interest in the feasibility of even lower precision training. The exponential demand for compute involved in training these deep neural networks has lead to multiple advancements in lower precision data types. Several studies have developed techniques such as loss scaling, stochastic rounding, and others to train effectively in 16-bit (; ; Azim), along with associated hardware support . Using 16-bit fixed point, showed that stochastic rounding techniques were crucial for model convergence even for simple convolutional neural networks. As noted in , Google's bfloat16 format has the same number of exponent bits as FP32, leading the success of that format without commonly requiring hardware intensive requirements such as stochastic rounding or other framework level techniques such as loss scaling. Although 8-bit formats have significant performance and memory advantages, convergence is especially challenging due to loss of accuracy in the backpropogated gradient values. demonstrated training models with matrix multiplications and convolutions in FP8 but they use FP16 with chunk-based accumulations and stochastic rounding hardware. also demonstrated success with FP8, accumulating in FP32 and using loss scaling techniques on ResNets, Transformer and GNMT networks. However, they too require the first and last layers of the model to be in FP32, and similar to leverage Stochastic Rounding techniques to maintain model accuracy. Unlike S2FP8 proposed in this work, both of these FP8 training techniques emphasize the need for efficient loss scaling, rounding hardware and restriction on some layers being in higher precision. quantized weights, activations and gradients of AlexNet to 1, 2 and 6 bits respectively. But they also need to maintain the first and last convolution layers in full precision and stochastically quantize the gradients. demonstrate using integers for training LeNet-5 and AlexNet with 8-bits for activations, error and gradients and 2-bits for weights. However, these approaches also required custom tuning such as novel initialization techniques and layer wise scaling instead of Batch Normalization and Softmax. These approaches lack generalizability to other models, requiring significant fine tuning. To the best of our knowledge, there does not exist an out-of-the-box solution using FP8 in training deep learning topologies without the need for tuned loss scaling techniques, requirements of certain layers being in full precision along with efficient hardware rounding schemes like Stochastic Rounding. 3 SHIFTED AND SQUEEZED 8-BIT FLOATING POINT FORMAT The FP8 format, with 2 bits of mantissa and 5 bits of exponent ) is both narrow (i.e., its dynamic range is very limited, from 2 −16 to 2 16) and has lower accuracy (the machine epsilon is only 2 −3). Figure A1 illustrates the range and accuracy of FP8. In contrast, FP32 ranges from 2 −149 to 2 128 with a machine-epsilon of 2 −24 (Table A1). On the other hand, tensors involved in neural networks (weights, activations and gradients) are spread across varying scales. As illustrated in Figure 1, the tensor distributions change over the course of training, spanning different orders of magnitude. As a , 8-bit training usually requires a combination of multiple techniques to capture the full dynamic range of values for model training. Some of these techniques include: • Loss scaling scales the loss L(w) by a constant λ before backpropagation. This makes the gradients artificially larger, allowing them to fit within the FP8 range. Gradients are then scaled down before being accumulated into the trainable weights as shown in Equation 6 • Stochastic rounding alleviate quantization errors by capturing some of the information discarded when truncating to lower precision at the output of a GEMM operation Between these two techniques, loss scaling is more critical; once the magnitude of the gradients can no longer be represented in the FP8 range, training convergence will not be possible. However, loss scaling only modifies the gradients. Weights and activations can also (albeit admittedly less frequently) exceed the FP8's representable range of [2 −16, 2 16]. In those scenarios, convergence can also be affected. The issue with loss scaling is that it requires user interaction. Models have to be modified, and, more importantly, tedious empirical tuning is required to find the correct loss scaling schedule. While some networks can be trained with constant loss scaling, some, notably Transformers, require dynamic "back-off" and improved loss scaling. This requires significant trial and error to tune the scaling schedule, slowing down wide adoption of low-precision numerical formats. To alleviate these issues and make neural network training possible with no model modifications or hyperparameter tuning, we propose a new 8-bit floating point format. Consider a tensor X of size N, i.e., X = {X i} N i=1. Instead of directly encoding each X i in FP8, we store X using N FP8 numbers accompanied by two (squeeze and shift) factors α and β (the "statistics" -see Figure 2). Figure 3: Impact of the Shifted and Squeezed transformation log 2 |Y | = α log 2 |X| + β. α let the distribution be as wide as necessary (though, with an associated loss of precision), and β let us shift the distribution around any value. For X i = 0, X and Y are then related through where the ± is chosen so that X i and Y i have the same sign. This representation allows for α and β be chosen so that together with tensor Y they capture most of the dynamic range of the tensor X. As we will see in section 4, this is all that is necessary to train networks using 8-bit floating point numbers. In order for Y to be a tensor suitable to be represented by FP8 numbers, we enforce that it has zero mean and a maximum value within the dynamic range of FP8 (e.g. 15): where the notation indicates that the sum and the max, respectively, ignore any i such that Y i = 0. Those equations ensure that log 2 (|Y |) values are distributed with zero mean and each is less than 15, which is ideal for an FP8 format. we find This new tensor format in the training procedure (forward pass, backward pass, weight update) described in Figure 4. Forward and backward MatMul use this new S2FP8 format. Master weights are kept in FP32 and updated using S2FP8 gradients. Accumulations inside the GEMM kernel are kept in full FP32 precision. Figure 3 illustrates the impact of α and β. By having those two extra degrees of freedom for each tensor, majority of the dynamic range of each tensor can now be captured, whether very small (β > 0), very large (β < 1), very narrow (α > 1)) or very wide (α < 1). One way to interpret α and β is to consider them as parameters of a distribution generating the tensor values log 2 (|X i |). We can then say that, by continuously computing α and β, we are effectively learning the distribution of log 2 (|X i |). Figure 5c shows the evolution of µ, m, α and β for a particular tensor of ResNet-20. We see that α and β converge to, approximately, 5 and 21, respectively. From Equation 1, we conclude that:, from FP32 to S2FP8. When using S2FP8 for training, forward and backward GEMM's only use S2FP8. The master weights are kept in FP32 and updated during the update step. • since α > 1, this means that X is expanded into Y, i.e., X is more narrow than what FP8 allows • since β > 0, this means that X is right-shifted into Y, i.e., X is smaller than what FP8 allows At convergence, those α and β values represent the distribution of each converged tensor. Notice that all statistics stabilize in the last third of the training, where the learning rate is decreased, indicating the network is converging to its final state. In this section, we compare S2FP8 training with baseline FP32 and FP8 training with and without loss scaling for: Residual Networks of varying depths on the CIFAR-10 and ImageNet datasets, Transformer on IWSLT'15 EnglishVietnamese dataset , and Neural Collaborative Filtering (NCF) on MovieLens 1 Million dataset . For our experiments, we use the open source Tensorflow Models 1 repository for ResNet and NCF, Tensor2Tensor for Transformer with added S2FP8 data type simulation support using the methodology described in subsection 4.1. For a given model, we keep the hyperparameters consistent across FP32, FP8 and S2FP8 evaluations. We simulated S2FP8 by inserting appropriate truncation function throughout the network, before and after every convolution and matrix-matrix product operations, during both the forward and backward passes. The rest of the network is kept in FP32, and those truncation simulate the low-precision training described in subsection 3.2. The truncation function takes as input a tensor X, computes its magnitude mean and maximum, computes the appropriate α and β and finally truncates X by computing where truncate FP8 is a usual FP8 truncation function with RNE (round-to-nearest, with ties broken by rounding to even) rounding which is easier to implement and most widely supported in hardware. We first present with Residual Networks of varying depths on the CIFAR-10 image recognition dataset. We trained the model on 1 GPU using standard parameters: 250 epochs, batchsize of 128, SGD with momentum of 0.9, initial learning rate of 0.1 decreased by a factor of 10 after epochs 100, 150 and 200. Table 1 and Figure A2 presents the . We observe that S2FP8 reaches almost exactly the FP32 baseline, sometimes even improving over it. Out-of-the-box FP8 does not converge and has very poor accuracy. Finally, FP8 with constant loss scaling of 100 (FP8+LS We also evaluate S2FP8 on the 1000 class ImageNet dataset. Here, we trained the network on 4 GPUs using standard parameters: 90 epochs, batchsize of 256, SGD with momentum of 0.9, initial learning rate of 0.1 decreased by a factor of 10 after epochs 30, 60, 80 and 90. Table 2 and Figure 6 present the . Again, we observe that S2FP8 gets very close to the FP32 baseline. Out-of-the-box FP8 quickly diverges and does not converge at all. For FP8 with loss scaling to converge, one has to not truncate the first and last layer, as consistent with, which we denote as Ex in Table 2 below. A loss scaling of 10,000 can then be used to reach the baseline (FP8+LS(10k)+Ex). Finally, stochastic rounding can be added and it slightly improves the precision (FP8+LS(100k)+Ex+SR). However, both those cases are not out-of-the-box, as they require loss scaling tuning and some layers to be kept in full precision. S2FP8 does not suffer from that, thanks to its improved quantization: all layers can be truncated and no loss scaling is required. We also tested S2FP8 on a small Transformer (Transformer Tiny) on the English-Vietnamese dataset. The model has 2 hidden layers of size 128, and a filter of size 512, and is trained using Adam optimizer . Table 3 and Figure 7 show the , where we compare FP32, S2FP8 and FP8 with exponential loss scaling. We tried many loss scaling schedules (constant and exponential, with various initializations) and report the best . As one can see, S2FP8 reaches the baseline with no hyperparameter tuning. FP8, on the other hand, does not, even after extensive loss scaling tuning. This shows the value of an out-of-the-box method for the user. En-Vi FP32 S2FP8 ∆ FP8 FP8+LS(exp) Transformer tiny 25.3 25.3 0.0 NaN 21.3 Table 3: BLEU Score (from 0 to 100) for translation task on the EnglishVietnamese dataset with Transformer tiny. The Neural Collaborative Filtering (NCF) network comprises of embeddings for users and items from the MovieLens dataset, that are then passed to a Multi-Layer Perceptron(MLP) network to learn the user-item interaction. Matrix-multiplication operations are the building blocks of such models. We compare S2FP8 with FP32 and FP8 without loss scaling. We simulate Matrix-Multiplications and look-ups from the embeddings in S2FP8 and compare it to FP8 with RNE. We trained the model on the MovieLens 1 Million dataset with the following standard paramaters: 20 iterations, batchsize of 1024 on 4 GPUs, 8 predictive factors, learning rate of 0.0005 using the Adam optimizer. Figure 8 and Table 4 show the , where we compare FP32, S2FP8 and FP8 without loss scaling. This again shows that S2FP8 easily reaches the baseline out-of-the-box, without tuning of any sort. FP8 gets relatively close, but cannot reach the baseline. S2FP8 is a new data type and requires its own circuitry to be implemented in a tensor processing engine. However, the added overhead is very minimal and affects neither data throughput nor compute speed. In order to convert FP32 tensors into S2FP8, two hardware (HW) components are needed. One is to calculate each tensor's statistics (Equation 3), which bring minimal HW complexity. To make compute operations even easier these statistics could be stored in lower precision such as FP8/INT8. The other component is to adjust the exponent and mantissa of all those tensor elements by applying the squeeze (α) and shift (β) factors in Equation 4 before truncating them into their 8-bit placeholders. The shift could be done using simple element-wise add/subtract operations on the exponents, and element-wise squeeze could be applied to the mantissa portions. Another consideration is within the tensor processing engine(e.g., GEMM engine) which requires the α and β factors while doing the calculations. The FP32 will be converted back to S2FP8 when needed (e.g., to store back in memory) as shown in Figure 4. We introduce a novel 8-bit floating point data type (S2FP8), that gives competitive performance in comparison to state-of-the-art FP32 baselines over a range of representative networks. S2FP8 makes use of shifted and squeezed factors to shift and rescale the range of tensors prior to truncation. S2FP8 allows training of neural networks with an 8-bit format while eliminating the need for loss scaling tuning, hardware-complex rounding techniques. In addition, compared to existing FP8 implementations we also eliminate the restriction of maintaining the first and last layers in FP32. Decreasing Movielens 1 million FP32 S2FP8 ∆ FP8 NCF 0.666 0.663 0.003 0.633 Figure A1: The range and precision of FP8. Bar indicate the number density between each power of 2. Since FP8 has 2 mantissa bit, the density is 4 (except in the denormals), and the associated machine epsilon is 2 −3 = 1/8. The normal representable range goes from 2 −14 to (1 − 2 −3)2 16, with denormals from 2 −16 to 2 −14. ∂(λL) ∂w (w) = λ ∂L ∂w (w) ⇒ w (k+1) = w (k) − α 1 λ ∂(λL) ∂w (w (k) ). Step L 2 Loss FP32 S2FP8 Figure A2: Convergence of ResNet-50 with the CIFAR-10 dataset

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel 8-bit format that eliminates the need for loss scaling, stochastic rounding, and other low precision techniques

Represent the TLDR for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Variational Auto Encoders (VAE) are capable of generating realistic images, sounds and video sequences. From practitioners point of view, we are usually interested in solving problems where tasks are learned sequentially, in a way that avoids revisiting all previous data at each stage. We address this problem by introducing a conceptually simple and scalable end-to-end approach of incorporating past knowledge by learning prior directly from the data. We consider scalable boosting-like approximation for intractable theoretical optimal prior. We provide empirical studies on two commonly used benchmarks, namely MNIST and Fashion MNIST on disjoint sequential image generation tasks. For each dataset proposed method delivers the best among comparable approaches, avoiding catastrophic forgetting in a fully automatic way with a fixed model architecture. Since most of the real-world datasets are unlabeled, unsupervised learning is an essential part of the machine learning field. Generative models allow us to obtain samples from observed empirical distributions of the complicated high-dimensional objects such as images, sounds or texts. This work is mostly devoted to VAEs with the focus on incremental learning setting. It was observed that VAEs ignore dimensions of latent variables and produce blurred reconstructions (; Sønderby et al., 2016). There are several approaches to address these issues, including amortization gap reduction , KL-term annealing and alternative optimization objectives introduction . In all cases, it was observed that the choice of the prior distribution is highly important and use of default Gaussian prior overregularizes the encoder. In this work, we address the problem of constructing the optimal prior for VAE. The form of optimal prior was obtained by maximizing a lower bound of the marginal likelihood (ELBO) as the aggregated posterior over the whole training dataset. To construct reasonable approximation, we consider a method of greedy KL-divergence projection. Applying the maximum entropy approach allows us to formulate a feasible optimization problem and avoid overfitting. The greedy manner in which components are added to prior reveals a high potential of the method in an incremental learning setting since we expect prior components to store information about previously learned tasks and overcome catastrophic forgetting (; ;). From practitioners point of view, it is essential to be able to store one model, capable of solving several tasks arriving sequentially. Hence, we propose the algorithm with one pair of encoderdecoder and update only the prior. We validate our method on the disjoint sequential image generation tasks. We consider MNIST and Fashion-MNIST datasets. VAEs consider two-step generative process by a prior over latent space p(z) and a conditional generative distribution p θ (x|z), parametrized by a deep neural network. Given empirical data distribution p e (x) = 1 N N n=1 δ(x − x n) we aim at maximizing expected marginal log-likelihood. Following the variational auto-encoder architecture, amortized inference is proposed by choosing variational c A. Authors. posterior distribution q φ (z|x) to be parametrized by DNN, ing in a following objective: To obtain a richer prior distribution one may combine variational and empirical bayes approaches and optimize the objective over the prior distribution. For a given empirical density, optimal prior is a mixture of posterior distributions in all points of the training dataset. Clearly, such prior leads to overfitting. Hence, keeping the same functional form, truncated version with K presudoinputs was proposed as VampPrior. In the present paper, we address two crucial drawbacks of the VampPrior. Firstly, for large values of K variational inference will be very computationally expensive. Even for the MNIST dataset used the mixture of 500 components. Secondly, it is not clear how to choose K for the particular dataset, as we have a trade-off between prior capacity and overfitting. For this purpose, we adapt maxentropy variational inference framework . We add components to the prior during training in a greedy manner and show that in such setting, fewer components are needed for the comparable performance. Assume, that we want to approximate complex distribution p * by a mixture of simple components p i Each component of the mixture is learned greedily. At the first stage we initialize mixture by a standard normal distribution. Afterwards, we add new component h from some family of distributions Q, with the weight α one by one p t = αh + (1 − α)p t−1, α ∈ in two stages: 1. Find optimal h ∈ Q. We apply Maximum Entropy approach to minimize KL-divergence between mixture and target distribution. This task can be reformulated as a following KL- 2. Choose α, corresponding to the optimal h: α * = arg min In this work, we suggest combining boosting algorithm for density distributions with the idea of VampPrior to deal with the problem of catastrophic forgetting in the incremental learning setting. Proposed algorithm for training VAE consists of two steps. On the first one we optimize evidence lower bound w.r.t. the parameters of the encoder and decoder. On the second stage, we learn new component h for the prior distribution, and its weight α, keeping parameters of the encoder and decoder fixed. We learn each component to be posterior given the learnable artificial input u: q φ (z|u) with target density being mixture of posteriors in all points from random subset M of the whoel training dataset D. Parameters of the first component u 0 are obtained by ELBO maximization simultaneously with the network parameters, as shown in the Algorithm 1. In the incremental learning setting, we do not have access to the whole dataset. Instead, subsets D 1,... D T arrive sequentially and may come from different domains. With the first task D 1 we Algorithm 1: BooVAE algorithm, λ, Maximal number of components K Output: p K, θ *, φ * Choose random subset M ⊂ D and initialize prior p 0 = q φ (z|u 0) and k = 1; θ *, φ *, u 0 = arg max L(p 0, θ, φ); follow Algorithm 1 to obtain prior p and optimal values of the network parameters. Starting from t > 1, we add regularization to ELBO, which ensures that the model keeps encoding and decoding learned prior components in the same manner (see Appendix B). Since we do not have access to the whole dataset anymore, the form of optimal prior also changes. We use prior of the previous step as a proxy for an optimal one (see Appendix C) p * (t) ≈ t−1 We perform experiments on MNIST dataset , containing ten hand-written digits and on fashion MNIST , which also has ten classes of different clothing items. Therefore, we have ten tasks in each dataset for sequential learning. To evaluate the performance of the VAE approach, we estimate a negative log-likelihood (NLL) on the test set, calculated by importance sampling method, with 5000 samples for each observation. In an offline setting, we aim at comparing our method to VampPrior and Mixture of Gaussians prior. In the first case, each component is a posterior distribution given learnable pseudo-input, while in the second we learn each component as a Gaussian with diagonal covariance matrix in the latent space. For both priors, all the component are learned simultaneously. Results in the tables above demonstrate that BooVAE manages to overcome catastrophic forgetting better than pure EWC regularization with standard normal prior. Unstable NLL values for VAE with standard prior and EWC can be explained by the fact that some classes in the dataset are quite similar and even though model forgets old class, knowledge about a new class let her reconstruct it relatively good, ing in deceptively better . In Appendix D we provide for each task separately to illustrate this effect further. Figure 1: Samples from prior after training on 10 tasks incrementally. Our approach is capable of sampling images from different tasks, while other methods either stick to the latest seen class, or are able to reproduce images from simplest classes On Figure 1, we provide samples from the prior illustrating generation ability of all the methods after training on all of the ten tasks sequentially. Appendix E and F provide detailed qualitative and quantitative (Figure 3) evaluation of the samples diversity for different models. In this work, we propose a method for learning a data-driven prior, using a MM algorithm which allows us to reduce the number of components in the prior distribution without the loss of performance. Based on this method, we suggest an efficient algorithm for incremental VAE learning which has single encoder-decoder pair for all the tasks and drastically reduces catastrophic forgetting. To make sure that model keeps encoding and decoding prior components as it did during the training of the corresponding task, we add a regularization term to ELBO. For that purpose at the end of each task we compute reconstructions of the components' mean value p θ j (x|µ i,j) = p ij (x). All thing considered, objective for the maximization step is the following: When training a new component and its' weight, we want optimal prior that we approximate to be close to the mixture of variational posteriors at all the training points, seen by the model. Since we don't have access to the data from the previous tasks, we suggest using trained prior as a proxy for the corresponding part of the mixture. Therefore, optimal prior for tasks 1: t can be expressed, using optimal prior from the previous step and training dataset from the current stage: During training, we approximate optimal prior by the mixture p (t), using random subset containing n observations of the given dataset M t ⊂ D t. Therefore, the optimal prior can be approximated in the following way: To evaluate diversity of the generated images, we calculate KL-divergence between Bernoulli distribution with equal probability for each class and empirical distribution of classes generated by the model. Since we want to assign classes to generated images automatically, we train classification network, which can classify images with high probability (more than 90%), use it to label generated objects and calculated the empirical distribution over 10000 generated samples. Figure 3 depicts proposed metric to evaluate diversity. We want this value to stay as close as possible to 0 as the number of tasks grows since it will mean that model keeps generating diverse images. We can see a drastic difference between boosting and other approaches: samples from prior, estimated by the boosting approach are very close to uniform in contrast to all the comparable methods.

Novel algorithm for Incremental learning of VAE with fixed architecture

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a framework for modifying the latent space operations such that the distribution mismatch between the resulting outputs and the prior distribution the generative model was trained on is fully eliminated.