title: File system causing c++ program to run slowly
body: <p>I am reading 1 million files from my local system (Ubuntu 20.04), about 10KB each. The first 40000 files are processed very quickly, and then the next 960000 are going at a much slower pace. I am assume this has something to do with the OS/Hardware/File System functionality. I am basically performing a file read as follows:</p>
  <pre><code>string read_file(std::ifstream&amp; ifs) {
       vector&lt;char&gt; buf(64 * 1024);
       string s = &quot;&quot;;
       ifs.read(&amp;buf[0], buf.size());
       while (int n = ifs.gcount()) { s.append(&amp;buf[0], n); ifs.read(&amp;buf[0], buf.size()); }
       return s;
  }
  </code></pre>
  <p>Can someone explain what is happening to cause this slow down and how it can be fixed?</p>
  <p>EDIT: I am processing the strings in a function, I am not sure if they persist after execution:</p>
  <pre><code> void process_file(string file_name) {
        ifstream ifs(file_name);
        string s = read_file(ifs);
        process_data(s);
   }
  
   void p1() {
        vector&lt;string&gt; files = get_files(); // 1 million size vector of filenames
        for (int i = 0 ; i &lt; files.size(); i++) {
             process_file(files[i]);
        }
   }
  </code></pre>
  <p>process_data writes to another file but does not keep the string</p>
  <p>EDIT: I have commented out the <code>process_data(s)</code> function and passed the <code>file_name</code> by reference and am getting the same behavior. However, if I run the program up to a number N, ctrl C, and then re-run, the first N files load very fast.</p>
  <pre><code> void process_file(string&amp; file_name) {
        ifstream ifs(file_name);
        string s = read_file(ifs);
        // process_data(s);
   }
  
   void p1() {
        vector&lt;string&gt; files = get_files(); // 1 million size vector of filenames
        for (int i = 0 ; i &lt; files.size(); i++) {
             process_file(files[i]);
        }
   } 
  </code></pre>