title: Logging lots of Android sensor data
body: <p>I have an Android application that is logging several Android sensors at approximately 100Hz. So if i am logging 10 sensors, I am writing about 3000 data points per second to a file (each sensor typically has 3 entries). Now the problem is that i want to minimize the effect of this writing on the rest of the app. Specifically, I do not want the logging to slow down event delivery... i want to make sure that i am getting events as soon as they happen, and not with a delay (i know there will always be some delay because Android is not realtime and because of the "pull" nature of the Sensor event framework). </p>
  
  <p>Next, I will outline my approach, which does not appear to be working well. I would like suggestions for how to improve. </p>
  
  <p>My current procedure is this... </p>
  
  <p>For each sensor, i create a separate thread with a BlockingQueue of events to log. Inside the thread, I have a while loop that pulls from the queue and does the file writing using a buffered writer. When the sensor manager delivers a new SensorEvent, the event is put in the appropriate queue (thereby triggering the file IO on the other thread) so as not to delay the  main thread on which SensorEvents are delivered.</p>
  
  <p>I want to be getting the events as soon as they occur, so it is important that i do not introduce any delays in the Sensor framework. If, for instance, I did the file IO directly in the onEvent callback, then i  am worried that events could start piling up in the pipeline, and that they would then be out of date by the time they are finally delivered. The above approach mitigates these worries. </p>
  
  <p>But there is another issue... </p>
  
  <p>Despite the fact that the file IO occurs off of the sensor event delivery thread, at times the application still feels sluggish. That is, sometimes i will see events occur in rapid succession (e.g. 5 events are delivered within 1 ms of each other). This indicates that although the IO is not happening on the sensor delivery thread, the delivery thread is still getting delayed. A few reasons have been suggested to me:</p>
  
  <ol>
  <li><p>I am creating too many IO threads. Perhaps if i pushed all of the writing to a single thread I would increase the likelihood that the sensor delivery thread is alive when a new event comes in. In the current setup, it could be that all of the active threads are being used for file IO when an Event comes in,  resulting in the events backing up until one of the writing events finishes.</p></li>
  <li><p>Currently, i am using flat file output, not a database. The benefits of using a database for retrieval are clear to me. What is not clear to me is whether i should also expect a database to be faster if I am only appending data to a file.... that is, i never need to read from the file or insert the data into a random place , I just literally append to the end of the file. It seems to me that a database cannot be any faster than standard file IO in that case. Or am I wrong?</p></li>
  <li><p>Others have suggested that the garbage collector is probably interfering with my threads and that the likely source of the problem is memory thrashing due to the large number of events that are being created.</p></li>
  </ol>
  
  <p>From which angle should i approach this?</p>
  
  <hr>
  
  <p>Edit: </p>
  
  <p>The following is the class I am using to write strings to file. One of them is created per SensorEvent type. </p>
  
  <pre><code>package io.pcess.utils;
  
  import java.io.BufferedWriter;
  import java.io.File;
  import java.io.FileWriter;
  import java.io.IOException;
  import java.util.concurrent.BlockingQueue;
  import java.util.concurrent.LinkedBlockingQueue;
  
  /**
   * This type of File Writer internally creates a low-priority {@link Thread}
   * which queues data to be written to a specified file. Note that the
   * constructor starts the {@link Thread}. Furthermore, you can then append a
   * {@link String} to the end of the specified file by calling
   *
   * &lt;pre&gt;
   * fileWritingThread.append(stringToAppend);
   * &lt;/pre&gt;
   *
   * Finally, you should tidy up by calling
   *
   * &lt;pre&gt;
   * fileWritingThread.close();
   * &lt;/pre&gt;
   *
   * which will force the writer to finish what it is doing and close. Note that
   * some {@link String}s might be left in the queue when closing, and hence will
   * never be written.
   */
  public class NonblockingFileWriter {
  
      /**
       * ---------------------------------------------
       *
       * Private Fields
       *
       * ---------------------------------------------
       */
      /** The {@link Thread} on which the file writing will occur. */
      private Thread                      thread     = null;
  
      /** The writer which does the actual file writing. **/
      private BufferedWriter              writer     = null;
  
      /** A Lock for the {@link #writer} to ensure thread-safeness */
      private final Object                writerLock = new Object();
  
      /** {@link BlockingQueue} of data to write **/
      private final BlockingQueue&lt;String&gt; data       = new LinkedBlockingQueue&lt;String&gt;();
  
      /** Flag indicating whether the {@link Runnable} is running. **/
      private volatile boolean            running    = false;
  
      /**
       * The {@link Runnable} which will do the actual file writing. This method
       * will keep writing until there is no more data in the list to write. Then
       * it will wait until more data is supplied, and continue.
       */
      private class FileWritingRunnable implements Runnable {
  
          @Override
          public void run() {
              try {
                  while (running) {
                      String string = data.take();
                      synchronized (writerLock) {
                          if (writer != null) {
                              writer.write(string);
                          }
                      }
                  }
              } catch (Exception e) {
                  e.printStackTrace();
              } finally {
                  close();
              }
          }
      };
  
      /**
       * ---------------------------------------------
       *
       * Constructors
       *
       * ---------------------------------------------
       */
      public NonblockingFileWriter(String filename) {
          this(new File(filename));
      }
  
      public NonblockingFileWriter(File file) {
          writer = createWriter(file);
          if (writer != null) {
              running = true;
          }
          thread = new Thread(new FileWritingRunnable());
          thread.setPriority(Thread.MIN_PRIORITY);
          thread.start();
      }
  
      /**
       * ---------------------------------------------
       *
       * Public Methods
       *
       * ---------------------------------------------
       */
      /** Append the specified string to the file. */
      public void append(String string) {
          try {
              data.put(string);
          } catch (InterruptedException e) {
              Thread.currentThread().interrupt();
          }
      }
  
      /**
       * Close the {@link BufferedWriter} and force the {@link Thread} to stop.
       */
      public void close() {
          running = false;
          try {
              synchronized (writerLock) {
                  if (writer != null) {
                      writer.close();
                      writer = null;
                  }
              }
              /**
               * This string will not be written, but ensures that this Runnable
               * will run to the end
               */
              data.put("Exit");
          } catch (Exception e) {
              e.printStackTrace();
          }
      }
  
      /**
       * Create a {@link BufferedWriter} for the specified file.
       *
       * @param file
       * @return
       */
      private BufferedWriter createWriter(File file) {
          BufferedWriter writer = null;
          if (!file.exists()) {
              try {
                  file.getParentFile().mkdirs();
                  file.createNewFile();
              } catch (IOException e) {
                  e.printStackTrace();
                  return writer;
              }
          }
          if (file.canWrite()) {
              boolean append = true;
              try {
                  synchronized (writerLock) {
                      writer = new BufferedWriter(new FileWriter(file, append));
                  }
              } catch (IOException e) {
                  e.printStackTrace();
              }
          }
          return writer;
      }
  
  }
  </code></pre>