title: using mmap() to search large file (~1TB)
body: <p>I'm working on a project that is trying to search for specific bytes (e.g. 0xAB) in a filesystem (e.g. ext2). I was able to find what I needed using <code>malloc()</code>, <code>realloc()</code>, and <code>memchr()</code>, but it seemed slow so I was looking into using <code>mmap()</code>. What I am trying to do is find a specific bytes, then copy them into a struct, so I have two questions: (1) is using <code>mmap()</code> the best strategy, and (2) why isn't the following code working (I get EINVAL error)?</p>
  
  <p>UPDATE: The following program compiles and runs but I still have a couple issues:
  <br>1) it won't display correct file size on large files (displayed correct size for 1GB flash drive, but not for 32GB)*.
  <br>2) it's not searching the mapping correctly**.</p>
  
  <p>*Is <a href="http://people.redhat.com/berrange/notes/largefile.html" rel="nofollow">THIS</a> a possible solution to getting the correct size using <code>stat64()</code>? If so, is it something I add in my Makefile? I haven't worked with makefiles much so I don't know how to add something like that. 
  <br>**Is this even the proper way to search?</p>
  
  <pre><code>#define _LARGEFILE64_SOURCE
  
  #include &lt;stdio.h&gt;
  #include &lt;fcntl.h&gt;
  #include &lt;stdlib.h&gt;
  #include &lt;string.h&gt;
  #include &lt;sys/stat.h&gt;
  #include &lt;sys/types.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;errno.h&gt; 
  #define handle_error(msg) \
  do { perror(msg); exit(EXIT_FAILURE); } while (0)
  
  int main(int argc, char **argv) {
  
      int fd = open("/dev/sdb1", O_RDONLY); 
  
      if(fd &lt; 0) {
          printf("Error %s\n", strerror(errno));
          return -1;
      }
  
      const char * map;   
  
      off64_t size;
      size = lseek64(fd, 0, SEEK_END);
      printf("file size: %llu\n", size);
      lseek64(fd, 0, SEEK_SET);    
  
      map = mmap(0, size, PROT_READ, MAP_SHARED, fd, 0); 
      if (map == MAP_FAILED) { handle_error("mmap error"); }
  
      printf("Searching for magic numbers...\n");
      for (i=0; i &lt; size; i++) {
      if(map[i] == 0X53 &amp;&amp; map[i + 1] == 0XEF) {  
          if ((map[i-32] == 0X00 &amp;&amp; map[i-31] == 0X00)  ||            
              (map[i-32] == 0X01 &amp;&amp; map[i-31] == 0X00)  ||
              (map[i-32] == 0X02 &amp;&amp; map[i-31] == 0X00)) {
              if(j &lt;= 5) { 
                  printf("superblock %d found\n", j);
                  ++j; 
              } else break;
  
      int q;
      for(q=0; q&lt;j; q++) {
          printf("SUPERBLOCK[%d]: %d\n", q+1, sb_pos[q]);
      }
  
      fclose(fd);
      munmap(map, size);
      return 0;
  }
  </code></pre>
  
  <p>Thanks for your help.</p>