Finding specific small byte arrays in large binary files I'm trying to quickly and efficiently find every recurring position of small byte arrays (4 bytes) in large binary files (several GBs). My current method is as follows:

Stream stream = File.OpenRead(filepath);

List&lt;long&gt; searchResults = new List&lt;long&gt;(); //The results as offsets within the file
int searchPosition = 0; //Track of how much of the array has been matched
int[] searchPattern = { 0x00, 0x01, 0x02, 0x03 }; // The array to search

while(true) //Loop until we reach the end of the file
{
    var latestbyte = stream.ReadByte();
    if(latestbyte == -1) break; //We have reached the end of the file

    if(latestbyte == searchPattern[searchPosition]
    {
        searchPosition++;
        if(searchPosition == searchPattern.Length)
        {
            searchResults.Add(stream.Position);
        }
    }
    else
    {
        searchPosition = 0;
    }
}


It's slow, and seems quite inefficient (3-4 seconds for a small 174MB file, 35 seconds for a 3GB one). 

How can I improve the performance? 

I looked into Boyer-Moore, but is it really worth it, considering the pattern i'm looking for is only 4 bytes?
You should be aware, that if you have a search pattern like:

byte[] pattern = { 0x00, 0x01, 0x02, 0x03 };


and a file sequence as 

..., 0x00, 0x00, 0x01, 0x02, 0x03,...


Then it won't be found, because the first 0x00 will increment searchPosition to 1, so the second 0x00 will test false in if(latestbyte == searchPattern[searchPosition]. 

The solution is to reset the searchPosition whenever the latestbyte equals searchPattern[0]:

while(true) //Loop until we reach the end of the file
{
    var latestbyte = stream.ReadByte();
    if(latestbyte == -1) break; //We have reached the end of the file

    if (latestbyte == searchPattern[0])
      searchPosition = 0;

    if(latestbyte == searchPattern[searchPosition]
    {
        searchPosition++;
        if(searchPosition == searchPattern.Length)
        {
            searchResults.Add(stream.Position);
            searchPosition = 0;
        }
    }
}


OBS: This is only working for search patterns with  bytes, that are all different (see Firdas comment).



The position of the found pattern, you add to the result is actually the position of the last byte in the pattern. Maybe the start position is more interesting: 

searchResults.Add(stream.Position - searchPattern.Length);



As Sylvain suggests you can improve performance a bit by reading larger chunks of bytes and evaluate them in turn successively:

List&lt;long&gt; GetPatternPositions(string path, byte[] pattern)
{
  using (FileStream stream = new FileStream(path, FileMode.Open))
  {
    List&lt;long&gt; searchResults = new List&lt;long&gt;(); //The results as offsets within the file
    int patternPosition = 0; //Track of how much of the array has been matched
    long filePosition = 0;
    long bufferSize = Math.Min(stream.Length, 100_000);

    byte[] buffer = new byte[bufferSize];
    int readCount = 0;

    while ((readCount = stream.Read(buffer, 0, buffer.Length)) &gt; 0)
    {
      for (int i = 0; i &lt; readCount; i++)
      {
        byte currentByte = buffer[i];

        if (currentByte == pattern[0])
          patternPosition = 0;

        if (currentByte == pattern[patternPosition])
        {
          patternPosition++;
          if (patternPosition == pattern.Length)
          {
            searchResults.Add(filePosition + 1 - pattern.Length);
            patternPosition = 0;
          }
        }
        filePosition++;
      }
    }

    return searchResults;
  }
}


The improvement is not that impressive, but maybe you can experiment with the bufferSize to optimize it?