title: C: Find and analyze clusters in a binary data stream
body: <h1>Structure of the file and description of the system</h1>
  
  <p>The stream I want to analyze (a large binary file) is composed as follows:</p>
  
  <ul>
  <li>40-bytes header</li>
  <li><p>A stream of 10-bytes signals:</p>
  
  <ul>
  <li>The first 8 bytes represent the time when the signal was registered</li>
  <li>The last 2 bytes describe the channel where the signal was registered</li>
  </ul></li>
  </ul>
  
  <p>The signal is emitted by a source which sends an impulse every <code>SIGNAL_INTERVAL</code>, and it may or may not be retrieved by the detector. If a detector counts, it sends the result to a counter's channel, which prints the count as shown above. The counter has 8 channels in total.</p>
  
  <h1>Multiplexing</h1>
  
  <p>In order to increase the number of detectors, a multiplexing approach is used. Two detectors send their counts to the same channel (say, detectors 1 and 9 are coupled on channel 1 of the counter). One of the signals (for example, 9) is delayed by <code>DELAY</code>, so that the delayed counts are shifted with respect to the non delayed ones.</p>
  
  <h1>Demultiplexing</h1>
  
  <p>The idea would be to divide the delayed data from the non delayed ones, then subtract the delay (adding 8 to the channel value so that a delayed count on channel 1 will be shown as a count on channel 9) and then rejoin the two arrays.</p>
  
  <p>If <code>SIGNAL_INTERVAL</code> is constant, this is relatively easy: I define a "mask" <code>[0, DELAY, SIGNAL_INTERVAL]</code> and, taking a reference timestamp value, see where every count falls in the mask.</p>
  
  <p>Trying different masks and counting which one gives the most counts, one can identify the delayed counts from the non delayed ones. This last part is done because we allow an error to the time count, so the stream will not be perfectly clustered. Moreover, it's impossible to know a priori if the first count is a delayed one, a non delayed one or even a casual count.</p>
  
  <p>This is done channel by channel, as the channels may have a different response time from each other.</p>
  
  <p>With this approach, the code is quite simple:</p>
  
  <pre><code>uint64_t maskCheck(struct count *data, int ch_num, int elements){
  
      const int MAX_NUM = 27; // Maximum number of masks checked
  
      int ref = 0; // The reference variable used as starting point at every cycle
      uint64_t sing_count[2][MAX_NUM]; // The array containing the singles counts
      uint64_t max_count; // Variable used to find the maximum in the array
      int t = 0; // Time index for the following loop
      uint64_t result = 0; // The final result, i.e. the index with the most singles counts
  
  // Initializing max_count (it has MAX_NUM as length, so it must be initialized after being declared)
      for(int i = 0; i &lt; MAX_NUM; i++)sing_count[0][i] = 0;
      for(int i = 0; i &lt; MAX_NUM; i++)sing_count[1][i] = 0;
  
      while(getChannel(data[t])!=ch_num){
          t++;
          if(t == elements - 1){
              printf("%s\n", "Nothing found");
              return 0;
          }
      }
      if(getChannel(data[t]) == ch_num) ref = getTimestamp(data[t]);
      uint64_t ref_indexed = ref;
  
      for(int index = 0; index &lt; MAX_NUM; index++){
  
  
          sing_count[1][index] = ref + nsToBins(index) - nsToBins(MAX_NUM/2);
          ref_indexed = sing_count[1][index];
  
          for(t = 0; t &lt; elements; t++){
          // Skips the counts not occurring at ch_num
              if(getChannel(data[t]) != ch_num) {
                  continue;
              }
  
  
          if(longAbs(getTimestamp(data[t]), ref_indexed) % nsToBins(SIGNAL_INTERVAL) &lt;= nsToBins(MASK) + nsToBins(COUNT_ERROR) &amp;&amp;
             longAbs(getTimestamp(data[t]), ref_indexed) % nsToBins(SIGNAL_INTERVAL) &gt;= nsToBins(MASK) - nsToBins(COUNT_ERROR)){
              sing_count[0][index]++;
  
          }
          else if(longAbs(getTimestamp(data[t]), ref_indexed) % nsToBins(SIGNAL_INTERVAL) &lt;= nsToBins(COUNT_ERROR) ||
              longAbs(getTimestamp(data[t]), ref_indexed) % nsToBins(SIGNAL_INTERVAL) &gt;= nsToBins(SIGNAL_INTERVAL) - nsToBins(COUNT_ERROR)){
              sing_count[0][index]++; 
          }
          }
      }
  
  // This last part maximizes the array.
      max_count = sing_count[0][0];
      result = sing_count[1][0];
  
      for(int i = 1; i &lt; MAX_NUM; i++){
          if(sing_count[0][i] &gt; max_count)
          {
              max_count = sing_count[0][i];
              result = sing_count[1][i];
          }
      }
  </code></pre>
  
  <p>where <code>struct count</code> is defined as a 10-byte array read by the functions <code>getTimestamp()</code> and <code>getChannel()</code>, and <code>nsToBins()</code> simply converts the time units.</p>
  
  <p>Having the "best mask", I can divide the array through it and then perform all the other needed operations.</p>
  
  <h1>The problem</h1>
  
  <p>Now, here comes the problem. <code>SIGNAL_INTERVAL</code> is not constant, and it's not even well determined (to give you an idea, the frequency oscillates between <code>75.6 Mhz</code> and <code>76.3 MHz</code>).
  The above approach turns out to be very unsuccessful now:</p>
  
  <ul>
  <li><code>SIGNAL_INTERVAL</code> has an error of about <code>0.3 ns</code></li>
  <li>The measurement is performed over 30 seconds</li>
  <li>Keeping in mind that the order of magnitude of <code>SIGNAL_INTERVAL</code> is of <code>10 ns</code>, after just one second the error would be too big</li>
  <li>This results in the timestamps being incorrectly divided, affecting all the subsequent operations.</li>
  </ul>
  
  <p>What I had in mind was something to analyze the clusters in the data (<code>SIGNAL_INTERVAL</code> is not constant, but the oscillation is much smaller than <code>DELAY</code> so some clustering could in principle be observed) and find another way to divide the two arrays.</p>
  
  <p>But so far I have nothing. Any help would be appreciated.</p>