Supporting programming on touchscreen devices requires effective text input and editing methods. Unfortunately, the virtual keyboard can be inefficient and uses valuable screen space on already small devices. Recent advances in stylus input make handwriting a potentially viable text input solution for programming on touchscreen devices. The primary barrier, however, is that handwriting recognition systems are built to take advantage of the rules of natural language, not those of a programming language. In this paper, we explore this particular problem of handwriting recognition for source code. We collect and make publicly available a dataset of handwritten Python code samples from 15 participants and we characterize the typical recognition errors for this handwritten Python source code when using a state-of-the-art handwriting recognition tool. We present an approach to improve the recognition accuracy by augmenting a handwriting recognizer with the programming language grammar rules. Our experiment on the collected dataset shows an 8.6 word error rate and a 3.6 character error rate which outperforms standard handwriting recognition systems and compares favorably to typing source code on virtual keyboards.
@cite_35: The handwritten script recognition problem is modeled in the framework of the hidden Markov model. For English text, which is the focus of the present research, the states can be identified with the letters of the alphabet, and the optimum symbols can be generated. In order to do so, a quantitative definition of symbols, in terms of features, is required. Fourteen features (some old, some new) are proposed for this task. Using the existing statistical knowledge about the English language, the calculation of the model parameters is immensely simplified. Once the model is established, the Viterbi algorithm is proposed to recognize the single best optimal state sequence, i.e. sequence of letters comprising the word. The modification of the recognition algorithm to accommodate context information is also discussed. Some experimental results are provided indicating the success of the new scheme. >
@cite_37: 
@cite_14: 
@cite_4: A handwritten script recognition system is presented which uses Hidden Markov Models (HMM), a technique widely used in speech recognition. The script is encoded as templates in the form of a sequence of quantised inclination angles of short equal length vectors together with some additional features. A HMM is created for each written word from a set of training data. Incoming templates are recognised by calculating which model has the highest probability for producing that template. The task chosen to test the system is that of handwritten word recognition, where the words are digits written by one person. Results are given which show that HMMs provide a versatile pattern matching tool suitable for some image processing tasks as well as speech processing problems.
@cite_41: Abstract Postal address interpretation is the task of assigning to letter mail pieces a delivery point encoding, e.g., ZIP+4 Code. The encoding is determined from images of destination addresses on mail piece faces; addresses that are handwritten, are of poor-quality machine printing, are incomplete or incorrect. This paper describes several recognition algorithms used in the interpretation of handwritten and machine-printed address text (digits symbols alphabets words).
@cite_29: This paper presents the online handwriting recognition system NPen++ developed at the University of Karlsruhe and Carnegie Mellon University. The NPen++ recognition engine is based on a multi-state time delay neural network and yields recognition rates from 96 for a 5,000 word dictionary to 93.4 on a 20,000 word dictionary and 91.2 for a 50,000 word dictionary. The proposed tree search and pruning technique reduces the search space considerably without losing too much recognition performance compared to an exhaustive search. This enables the NPen++ recognizer to be run in real-time with large dictionaries. Initial recognition rates for whole sentences are promising and show that the MS-TDNN architecture is suited to recognizing handwritten data ranging from single characters to whole sentences.
@cite_6: 
@cite_43: The paper presents new A2iA bank check recognition systems designed to process handwritten and or printed checks issued in France, UK or USA. All the systems have identical architecture and design principles. However, each of them contains a country-specific part and is trained with country-specific data. Each system performs location, extraction, segmentation and recognition of courtesy and legal amounts in a document image, as well as deciding whether to accept or reject the check. The recognition rate is 80-90 . In the production mode, the check acceptance rate is 60-75 , with the misread rate corresponding to that of a human operator (close to 1 ).
@cite_40: This survey describes the state of the art of online handwriting recognition during a period of renewed activity in the field. It is based on an extensive review of the literature, including journal articles, conference proceedings, and patents. Online versus offline recognition, digitizer technology, and handwriting properties and recognition problems are discussed. Shape recognition algorithms, preprocessing and postprocessing techniques, experimental systems, and commercial products are examined. >
@cite_13: Hidden Markov model (HMM) based recognition of handwriting is now quite common, but the incorporation of HMM's into a complex stochastic language model for handwriting recognition is still in its infancy. We have taken advantage of developments in the speech processing field to build a more sophisticated handwriting recognition system. The pattern elements of the handwriting model are subcharacter stroke types modeled by HMMs. These HMMs are concatenated to form letter models, which are further embedded in a stochastic language model. In addition to better language modeling, we introduce new handwriting recognition features of various kinds. Some of these features have invariance properties, and some are segmental, covering a larger region of the input pattern. We have achieved a writer independent recognition rate of 94.5 on 3,823 unconstrained handwritten word samples from 18 writers covering a 32 word vocabulary.
Research in handwriting recognition has a long history dating back to the 1960s @cite_40 . Hidden Markov Model (HMM) based handwriting recognition @cite_13 @cite_35 @cite_4 is one of the most widely used approaches while neural networks are gaining in popularity @cite_29 . Some approaches also leverage additional constraints for recognizing handwriting in specific domains such as postal addresses @cite_41 @cite_6 and banking checks @cite_43 @cite_37 . These handwriting recognition systems are developed to take advantage of the English language @cite_14 , which is intrinsically different from source code. For instance, variable names are often created from concatenated words (e.g. camelCase or underscore naming), which poses a problem for the traditional handwriting recognition system as it expects spaces to appear between words contained within its dictionary. We do not aim to contribute to the extensive literature in handwriting recognition, but rather, we intend to examine how we can leverage this existing work for application to handwritten source code recognition interfaces.