text: In this article we will go through the initial creation steps of a touch enabled numerical computing environment, i.e., one that is optimized for touch enabled computing devices. The app is called Sumerics, which is short for sensor numerics. The core of the program is the Open Source project YAMP, which is basically a (highly extensible) parser and interpreter for numerical mathematics.
  The application has two main goals:
  We would like to do as much as we can with sensor data, however, that should not limit the user to such data. The user should be able to bring in his own data (from already saved binary files, images, plain text files, or text files with a CSV format), modify it, and make calculations with it. The application should never dictate the user to take sensor data, even though that will be one of the features of this application.
  With the application we want to present real-time visualization for the data. If a user selects one scalar we either show its point location in a Gaussian plane along with some statistics like absolute value and phase (for a complex number) or some interesting properties for an integer, like the previous and next prime number, its prime factors or the nor in general the numbers importance in the current workspace. If the user selects a matrix or vector we either display a histogram with the absolute values, or a 2D or 3D plot depending on the dimensions. If two saved variables are marked the application tries to find some relations between them. In some cases the most likely relation is a x-y plot. Otherwise a histogram is shown as well.
  The visualization does not stop here. We use the device orientation and the ambient light sensor to embed the 3D plots into the real environment of the device. The best thing, however, is that all plots can be generated in real time. This sounds like no usage at all, at first, but we will explain the graphical function editor later on, which can produce output, which can then be used for a final evaluation in form of a (3D) plot.
  We will discuss the required steps to write a plugin for YAMP that extends the given function base with special sensor functions. These functions are directly connected to the hardware sensors of the system / Ultrabook, via the Sensor API for desktop applications.
  Let's discuss the sensors we take care of first:
  acc()
  light()
  comp()
  orient()
  gyro()
  inc()
  gps()
  The function call with no parameter always returns the current value(s). Some functions might have more parameters, to switch some mode or return only a subset of possible return values. We implemented all functions in a way, that they are already documented and provide examples to the end-user. Therefore YAMP will do the required work for us, to display the appropriate help when the help() function has been used.
  Another important part of Sumerics is the user interface. Actually the user interface is what makes Sumerics different from other educational / numerical applications. On the one side we did not want to re-invent the wheel, on the other side it was quite important to use some oldies but goldies. Therefore we also use the three corner stones:
  Apart from this selection of oldies we added a new function box, with the frequently / recently used functions shown on top. The list can be easily searched or filtered (same is true for the workspace). Additionally we added a view with the available sensors, which simplifies access and shows the user the current values in nice real time charts.
  For giving our application some depth (while not ignoring the touch input possibilities), we added a panorama control. From this point on we can easily extend our applications with a lot of extra gadgets, views and possibilities. One of those possibilities is the graphical function editor, which will be discussed later on.
  Recently the article about the open source project YAMP has been published (available on the CodeProject). The project is currently in a pre-release state, such that some features are still missing or incomplete, however, since we are responsible for both, Sumerics and YAMP, a feature that is required for Sumerics and should be included in the core of YAMP, can be implemented right away.
  YAMP was created with two basic ideas in mind:
  The second idea evolved into creating a product for the latest Intel Ultrabook generation, that can be used with touch input, the mouse and / or the keyboard. At first we will not include the possibility for a pure command line interaction, however, this may come in the future.
  The final product will not only display sensor data and include the information of various sensors by functions, it will also benefit from the sensors in different views. Obviously the orientation aspect is used for determining the best size and position of the window. Accelerometer and orientation as well as the ambient-light-sensor will be used for displaying plots in a fresh, impressive way.
  Let's have a look at the big picture first (green are third party libraries, orange our own external libraries and blue the new libraries / application):
  So we create a WPF application called Sumerics, write two libraries which make use of YAMP, with one of them also using the different sensors on the Ultrabook. The application then makes use of YAMP and connects the plugins to YAMP, which enable users to make more out of YAMP.
  Therefore the architecture of Sumerics heavily relies on YAMP. However, some functions of YAMP should be easier to write for end-users than the pure (core) YAMP provides. Let's see an example of such a call. If the user wants to clear certain variables, he or she will have to enter the following command:
  clear("a", "b", "c")
  This clears 3 variables a, b and c. Sumerics now makes it possible to use the following command:
  a
  b
  c
  clear a b c
  This is working because Sumerics has a command parser as well. If the command parser finds a valid command, Sumerics will execute it. Otherwise the whole expression will be passed to YAMP, which will then parse and interpret the expression. The built-in commands can be anything from changing the GUI (like clearing the command history) to wrapping YAMP functions.
  Writing such a command parser is quite easy and can be done in no time. All we need is an abstract base class. In this case we create one that is named YCommand. We will make several implementations of YCommand, depending on our needs. Finally we will write a method that uses reflection to register the methods that our pre-parser (or let's call it command parser) can use.
  YCommand
  The register method looks as simple as the following code snippet:
  public static void RegisterCommands()
  {
  var lib = Assembly.GetExecutingAssembly();
  var types = lib.GetTypes();
  foreach (var type in types)
  {
  if (type.IsAbstract)
  continue;
  if (type.IsSubclassOf(typeof(YCommand)))
  {
  var cmd = type.GetConstructor(Type.EmptyTypes).Invoke(null) as YCommand;
  commands.Add(cmd.Name.ToLower(), cmd);
  }
  }
  }
  A small subset of the classes that make some functions available from the commands parser is displayed below.
  The command parser works with an arbitrary number of arguments. The basic code is quite similar to the one provided in the article about YAMP, where the ArgumentFunction class provides similar methods.
  ArgumentFunction
  For writing the specific sensor functions in our YAMP plugin, we first start with an abstract base class. This class will simplify our coding, since it reduces writing the same code over and over again. Also we just have to extend its functionality to supply all derived classes with more
  possibilities. Let's have a look at the implementation of the acc() function.
  using YAMP;
  using Microsoft.WindowsAPICodePack.Sensors;
  namespace YAMP.Sensors
  {
  public class AccFunction : SensorFunction
  {
  private static Accelerometer3D _accelerometer;
  static AccFunction()
  {
  if (!Manager.Initialized || _accelerometer != null)
  return;
  try
  {
  SensorList<Accelerometer3D> sl = SensorManager.GetSensorsByTypeId<Accelerometer3D>();
  if (sl.Count > 0)
  {
  _accelerometer = sl[0];
  }
  }
  catch (SensorPlatformException e)
  {
  System.Diagnostics.Debug.WriteLine(e.Message);
  }
  Exists = _accelerometer != null;
  }
  public static AccelerometerReading Measurement
  {
  get
  {
  if (!Manager.Initialized || !Exists)
  return null;
  AccelerometerReading reading = new AccelerometerReading(
  _accelerometer.CurrentAcceleration[AccelerationAxis.XAxis],
  _accelerometer.CurrentAcceleration[AccelerationAxis.YAxis],
  _accelerometer.CurrentAcceleration[AccelerationAxis.ZAxis]);
  return reading;
  }
  }
  public MatrixValue Function()
  {
  if (!Manager.Initialized || !Exists)
  return new MatrixValue(3, 1);
  var reading = Measurement;
  var result = new MatrixValue(3, 1);
  result[1].Value = reading.AccelerationX;
  result[2].Value = reading.AccelerationY;
  result[3].Value = reading.AccelerationZ;
  return result;
  }
  }
  public class AccelerometerReading
  {
  public double AccelerationX { get; private set; }
  public double AccelerationY { get; private set; }
  public double AccelerationZ { get; private set; }
  public AccelerometerReading(double AccelerationX, double AccelerationY, double AccelerationZ)
  {
  this.AccelerationX = AccelerationX;
  this.AccelerationY = AccelerationY;
  this.AccelerationZ = AccelerationZ;
  }
  }
  }
  Here we provide a lot of static functions, so that access directly from Sumerics is also possible (i.e. for live data of the sensors), without creating an explicit instance or having YAMP to interpret a fixed expression. We pack the reading of the sensor in a small nested class, AccelerometerReading, which is then returned by getting the current measurement.
  AccelerometerReading
  While in case of a non-existing sensor a null value is returned, YAMP will always return data. In such cases YAMP will return scalars / matrices with the correct dimensions, but without any values (so every value will be 0).
  null
  Handling the whole sensor stuff was pretty neat. At first everything looked amazingly simple, however, that was before I realized that the sensor API is (managed) only available in Windows Store apps. Therefore we have to use a wrapper around the unmanaged code, if we really want to use them (use we want to!) in C#.
  Writing the graphical function editor is quite a challenge, since it needs to be flexible and lightweight, but also powerful and robust. Here we have to add possibilities for using the keyboard:
  We also have to wire up events for the mouse. For this input device we will have to handle the following list of actions:
  Our application is fully touch enabled. The graphical editor was one of the center pieces of Sumerics, which resulted in a lot of different gestures being required to be implemented here.
  All those touch events must be wired up exactly and should not be counter-productive, but rather be optional additive, i.e. the user-experience does not rely on multiple input devices, but gets better with multiple (different) ones (instead of getting worse).
  We plan to include a lot of sample worksheets (written in YAMP script), to include out of the box working experiments with the sensors. It is easy to imagine that Sumerics will be able to do some "live experiments" without actually requiring to set up any components - just out of the box and with an Ultrabook. Therefore we are
  targeting educational institutions like higher level Physics courses for schools or experiments for undergraduates.
  Let's have a look at some items on the list of planned experiments:
  User interfaces like MATLAB, Mathematica and other scientific programs have issues with touch screens and do not take any sensor data into account. With YAMP we try to make a more modern UI, that extends the possibilities of a standard PC by giving users a unique experience with touch and a more extended experience with more interactions.
  Our target audience is currently busy as a researcher or student in higher education or at the University. The main purpose of Sumerics is to do numerical experiments with either numerical mathematics or using the provided sensor data to create experiments that work out of the box.
  Our target audience is highly tech-aware and will probably use an Ultrabook as an everyday gadget, to visualize things, create things and be creative. It is therefore possible, that Sumerics can easily support more sensors by just buying some hardware and plugging that in. Probably we are going to write a really generic sensor function, that can be accessed quite close to the driver.
  Additionally to the sensors that are unique about the Ultrabook, we are going to include some other sensors, which might give interesting data and can be used for some wicked experiments.
  First of all let's have a look at the "obvious" sensors that are going to be included:
  The moue can be used for experiments that are pulling something mechanically. A simple function (with help of user input) can roughly find out what the px / cm ratio is. With this knowledge you can estimate the velocity (if you know px / s and px / cm you can find out what cm / s is, which gives you m / s and km / h etc.) of the mouse.
  The touch input could be used to demonstrate how the touch signal works. It also allows the user to find out more about the properties of his or her touch screen.
  Finally we will also include the sound port as a sensor. The reasons for this are:
  Talking to some guys from the local electronics lab this sounds (!) like a great idea. Basically you can do a lot of useful things with the sound input. Connecting it (via functions) to the other sensors can create a powerful basis for (virtual as well as real) experiments. One possible thing is: measuring (via peak power spectrum of the sound input and the accelerometer) the relation between screaming and the current G-force on a roller-coaster ride.
  There are a lot more possibilities with this concept. We will include some samples in an included (with the release of Sumerics) collection of worksheets.
  The application requires an Ultrabook from Intel to be tested. To say the least I am really hyped and can't wait to get an Ultrabook in order to give this (in my opinion amazing) application the chance of becoming really robust.
  The testing will focus on the user experience in general and on getting the right sensor data. It will also focus on different test cases, i.e. what if a sensor does not return valid data or if a sensor is not available.
  The competition seemed to have enough time, but an aweful lot of work to deal with in November (and some interesting conferences) crossed my plans quite hard. Nevertheless, by coding 3 days in a row for about 16 hrs per day I was able to submit the app in time.
  Therefore I was quite relieved once I've seen the following part of my screen:
  That being said, let's dive in to get to know Sumerics -- the current state and the future plans.
  The screenshots that have been submitted to the Intel AppUp store show my design goals. The main view (called interaction) is where the console sits and most user interaction is
  happening.
  Of course this view (would) heavily favor keyboard interaction, which is why there are two key things in there. The first is an integrated on-screen keyboard, that slides-up when the user presses the keyboard button. This keyboard has just this one mode, which is enough to enter any statement. Those statements can include matrices, imaginary numbers (therefore the i has been placed on the keyboard next to the numbers), and all possible operators.
  The second thing is the Draw Expression button, which opens Microsoft's Math Input panel. Once the user has finished drawing the expression Sumerics is able to get
  the expression and enter it in the console. Sumerics can even handle the input of matrices and fractions without any problem.
  The second screenshot also shows the filtering of the functions. At the moment those functions are just documentation, but in the future the might be also good for drag and drop or other useful operations. This way the user has to type less (with an on-screen or real keyboard).
  A really important section is the plot section. Here the latest (selected or created, whatever was the last action) plot is shown. All modifications can be done in the console mode, however, in order to stay touch friendly the most common settings have been made available for GUI and touch users. All those settings can be accessed by pressing the buttons on the bottom (command) bar.
  The user can export the plot by saving it as a PNG in a file. He can also print out the shown plot or modify the included series' properties or generally applied settings. For convenience the gridlines switch has been created as a button. With the center button a plot can be reset to its default state. This is important since the plot can be zoomed and panned by touch manipulation. The last button opens a mini-console, which is just a small input window. This input window can be used to set other settings manually or create whole plots. The purpose of this dialog was to minimize the way for the user between console input and plot output.
  The graphical scripting is something that had to be left out, however, just for the moment. A big part is already implemented, but some things are still missing and a lot of testing still needs to be done. The scripting would look like the following screenshot:
  In this view the user basically (over touch or mouse) drags and drops a category of elements. Afterwards he is able to adjust the element (in this screenshots we have indivdual statement boxes placed - those have just a textbox. In general a combobox or some more advanced dialog are visible).
  The top nodes are input and output elements. In the screenshot no input elements have been added (they would be violet and on the left side) resulting in a function without any arguments. The chain consists of two statement blocks, the first without any input that gives its output to the second box. This second statement box directs its output to the output of the scripted function. In this case we would just get 13 * 29.
  We also created a small about dialog, which contains the user's GPS position (as an easter egg). Usually the compass and the GPS data has been located on the sensor plots section as well - but since the GPS data changes very slowly, placing it in the about dialog made much more sense.
  The about dialog shows the GPS and the compass data. In the latest version the data is also bound to the current state (which is updated regularly) of the GPS and the compass sensors.
  The icons for functions, matrices, etc. have been created on my own. All the other icons have been taken from the amazing Modern UI icons collection created by Austin Andrews. There are several hundred icons, which should be sufficient for most use-cases. The app theme was provided by the MAHApps Metro interface.
  Most controls have been taken from this library as well. However, neither MAHApps Metro nor the .NET-Framework includes a nice touch-friendly color picker that is suitable for WPF. Therefore I just wrote my own. I also searched for a nice console control, but I could not find anything that would be suitable. Therefore I ended up taking the Fast Colored TextBox for Syntax Highlighting by Pavel Torgashov and modifying it. Actually I removed a large part of the code and specialized it for my purposes. The main rendering of the control is still done in GDI+ since I figured out that porting it is probably possible, but might require too much time. I also completely re-wrote the autocomplete part - this time in WPF. The new code is about only a 20th or so and renders completely in WPF.
  The code consists of several async statements using the power of C# 5. In order to fully utilize the mighty WPF binding engine only the MVVM principle was used in the whole app, except the custom console. This control is actually a Windows Forms one, i.e. a class derived from a Windows Forms Control class with custom (GDI+) drawing. In the end using an existing (in this case Windows Forms) solution and modifying it turned out to be much faster than starting from scratch here!
  async
  Control
  The following (main) view models have been created:
  Several other (tiny) view models have also been implemented. Those should just support controls, dialogs or bigger view models. In the end it turned out to be the right solution, since we could archieve the following extension by just one copy and paste: Duplicating the variables bar in two different tabs.
  This does sound trivial, but it is actually a good test if a application has been well-designed from a programmer's perspective. Now everything seems to be in sync - as it should be.
  All in all Sumerics performs quite well and displays even large plots quite fast. This is because the guys from the open source project Oxyplot did an amazing job. The only thing that is not excellent is their implementation of panning and zooming. I thought they did not include something - but they did. The only negative thing about it is that it performs really bad. Zooming and panning takes a few seconds up to half a minute (depending on the data size). This is much longer than repainting the whole thing with changed axis. I searched for a possibility to prevent the in-built panning and zooming (and implement my own), but I could not find such a possibility. Therefore I ended up just hoping for an update from them, which solves this problem.
  In the following I will write a few lines about some interesting parts in the code. They are not revolutionary or required in any way, but they made programming the app a lot easier. The following code snippet has been used on various occasions:
  public static T GetWindow<T>() where T : Window, new()
  {
  foreach (Window window in App.Current.Windows)
  {
  if (window is T)
  {
  window.Activate();
  return (T)window;
  }
  }
  var win = new T();
  win.Show();
  return win;
  }
  So basically one can use this method to get an arbitrary Window. If the dialog is already open it will be brought to the foreground. Otherwise a new instance will be created and shown.
  Window
  Quite often you want to trigger something in XAML based on a property. Sometimes you are doing this with binding in mind - but what if the value you are binding to can be null? And furthermore what if you want to actually consider your trigger to start if the value is (or conversely is not) null? The following converter is the key:
  public class IsNullConverter : IValueConverter
  {
  public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
  {
  return value == null;
  }
  public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
  {
  throw new InvalidOperationException("IsNullConverter can only be used OneWay.");
  }
  }
  Using the converter in XAML is more or less straight forward. One just needs the following lines (assuming you are using local as the XML namespace for your "real" (i.e. C#) namespace:
  local
  <!-- In the resources (e.g. use in app.xaml or elsewhere) -->
  <ResourceDictionary>
  <local:IsNullConverter x:
  </ResourceDictionary>
  <!-- Now use a DataTrigger like this -->
  <DataTrigger Binding="{Binding WhatAreYouBindingTo?, Converter={StaticResource inc}}" Value="True">
  <!-- Place what should happen here ... -->
  </DataTrigger>
  To test some keys on various occasions I required a lot of duplicate code. I was not happy with the situation and decided to write some extension methods. The outcome looks like the following:
  public static bool IsCtrl(this KeyEventArgs keyEvent, Key value)
  {
  return keyEvent.KeyboardDevice.Modifiers == ModifierKeys.Control && keyEvent.Key == value;
  }
  public static bool IsModified(this KeyEventArgs keyEvent, Key value)
  {
  return keyEvent.KeyboardDevice.Modifiers != ModifierKeys.None && keyEvent.Key == value;
  }
  public static bool IsCtrlShift(this KeyEventArgs keyEvent, Key value)
  {
  return keyEvent.KeyboardDevice.Modifiers == ModifierKeys.Control && keyEvent.KeyboardDevice.Modifiers == ModifierKeys.Shift && keyEvent.Key == value;
  }
  public static bool Is(this KeyEventArgs keyEvent, Key value)
  {
  return keyEvent.KeyboardDevice.Modifiers == ModifierKeys.None && keyEvent.Key == value;
  }
  This allows me to directly test some KeyEventArgs object with a specific key considering a specific modifier.
  KeyEventArgs
  Also another code snippet worth sharing is the part where the MathML from the Windows Math Input Panel gets discovered and pasted right away. This has been achieved with the help of a clipboard handler. Unfortunately this is not built into the .NET-Framework (I actually do not know why - I think this is a really useful method!).
  //Yes we will need those pointers ...
  IntPtr viewerHandle = IntPtr.Zero;
  IntPtr installedHandle = IntPtr.Zero;
  //Two constants from the Win32 API
  const int WM_DRAWCLIPBOARD = 0x308;
  const int WM_CHANGECBCHAIN = 0x30D;
  //Get the required API calls
  [DllImport("user32.dll")]
  private extern static IntPtr SetClipboardViewer(IntPtr hWnd);
  [DllImport("user32.dll")]
  private extern static int ChangeClipboardChain(IntPtr hWnd, IntPtr hWndNext);
  [DllImport("user32.dll", CharSet = CharSet.Auto)]
  private extern static int SendMessage(IntPtr hWnd, int wMsg, IntPtr wParam, IntPtr lParam);
  protected override void OnSourceInitialized(EventArgs e)
  {
  base.OnSourceInitialized(e);
  var hwndSource = PresentationSource.FromVisual(this) as HwndSource;
  if (hwndSource != null)
  {
  installedHandle = hwndSource.Handle;
  viewerHandle = SetClipboardViewer(installedHandle);
  hwndSource.AddHook(new HwndSourceHook(this.hwndSourceHook));
  }
  }
  protected override void OnClosing(System.ComponentModel.CancelEventArgs e)
  {
  ChangeClipboardChain(this.installedHandle, this.viewerHandle);
  int error = System.Runtime.InteropServices.Marshal.GetLastWin32Error();
  e.Cancel = error != 0;
  base.OnClosing(e);
  }
  protected override void OnClosed(EventArgs e)
  {
  this.viewerHandle = IntPtr.Zero;
  this.installedHandle = IntPtr.Zero;
  base.OnClosed(e);
  }
  IntPtr hwndSourceHook(IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)
  {
  switch (msg)
  {
  case WM_CHANGECBCHAIN:
  this.viewerHandle = lParam;
  if (this.viewerHandle != IntPtr.Zero)
  SendMessage(this.viewerHandle, msg, wParam, lParam);
  break;
  case WM_DRAWCLIPBOARD:
  ReadClipboardData();
  if (this.viewerHandle != IntPtr.Zero)
  SendMessage(this.viewerHandle, msg, wParam, lParam);
  break;
  }
  return IntPtr.Zero;
  }
  void ReadClipboardData()
  {
  //MathML could be something else ...
  var stream = Clipboard.GetData("MathML") as Stream;
  if (stream != null)
  {
  //Do something with the stream!
  }
  }
  We are mainly looking for those pieces of MathML. Once we detect something we read the clipboard and see if it is actually of type MathML. If this is the case then we are either only pasting it directly or running the query as well. This is just a matter of setting Sumerics. The default option is to run the query once it gets pasted.
  I rushed to submit the app before the deadline, but I had bigger plans than the initial release. However, even though I could not implement all features before the 1st of December, I could implement some very important ones, which have been left out in the initial release. These features made it to version 0.9.8 (0.9.5 was the version that has been published initially) and have been submitted for validation on the 28th of November. Unfortunately I doubt that those updates will make it in time. This is actually a shame since this means that my competition submission will miss the following features:
  Now it's time to talk about the price tag on Sumerics. Actually it is for free at the moment. But this will probably not last. Once Sumerics reaches version 1.0.0 I will must probably start to sell it for like EUR 4.99 or something similar (definitely not over 5 euros). I have several reasons for a pricing like that. First and most important: The project was never meant to make me rich. Second: If the app is not for free then customers can already assume that there will be some support - so this makes sense for me. I'll fix some bugs and integrate features if people are willing to pay for this.
  And the last argument is actually a really nice one: There are some mathematics applications out there (on the AppUp store), which are bad (I mean: really bad!) and do cost more money. I will certainly proof the point that you can create a better app and make it even cheaper. Also I want to stay cheaper than the main competitors, which is why I will have to look at those (the possible competition) first.
  The screenshot above is showing the console in the latest version. We can see the autocomplete menu open and the syntax highlighting. We can also see that big matrices are folded by default. This improves the overview and guarantees that only the required (or short) output is displayed directly.
  We also see the commands in action. Some functions have been transformed to commands (by Sumerics). This is why instead of cd("..") the user can also enter cd ... The same is true for pwd() and dozens of other functions.
  cd("..")
  cd ..
  pwd()
  Here are some nice.