title: PHP - How to extract blocks from a text file by reading it line-by-line
body: <p>I have an input text file like the following:</p>
  
  <pre><code>BEGIN
  #1 
  #2 
  #3 
  #4 
  #5 
  #6 
  1       2015-05-31  2001-11-24  'Name Surname'      ID_1        0 
  2       2011-04-01  ?           ?                   ID_2        1 
  2       2013-02-24  ?           ?                   ID_3        1 
  2       2014-02-28  ?           'Name Surname'      ID_4        2 
  END
  #7      'value 1'
  #8      'value 2'
  #9      'value 3'
  #10     'value 4'
  END
  </code></pre>
  
  <p>When in the text file there is a <code>BEGIN</code>, from there starts a loop where each line that begins with <code>#</code> is a key, while the relative values are the columns of each following rows, until the <code>END</code>, generating arrays like the following:</p>
  
  <pre><code>Array ( [#1] =&gt; Array ( [0] =&gt; 1 [1] =&gt; 2 [2] =&gt; 2 [3] =&gt; 2 ) [#2] =&gt; Array ( [0] =&gt; 2015-05-31 [1] =&gt; 2011-04-01 [2] =&gt; 2013-02-24 [3] =&gt; 2014-02-28 ) [#3] =&gt; Array ( [0] =&gt; 2001-11-24 [1] =&gt; ? [2] =&gt; ? [3] =&gt; ? ) [#4] =&gt; Array ( [0] =&gt; 'Name Surname' [1] =&gt; ? [2] =&gt; ? [3] =&gt; 'Name Surname' ) [#5] =&gt; Array ( [0] =&gt; ID_1 [1] =&gt; ID_2 [2] =&gt; ID_3 [3] =&gt; ID_4 ) [#6] =&gt; Array ( [0] =&gt; 0 [1] =&gt; 1 [2] =&gt; 1 [3] =&gt; 2 ) )
  </code></pre>
  
  <p>Otherwise, if in the text file there isn't a <code>BEGIN</code>, but you find a line that starts with <code>#</code>, its relative value is the one between the single quotes, generating an array like the following:</p>
  
  <pre><code>Array ( [#7] =&gt; 'value 1' [#8] =&gt; 'value 2' [#9] =&gt; 'value 3' [#10] =&gt; 'value 4' )
  </code></pre>
  
  <p>This is what I would to obtain, and my current code is the following:</p>
  
  <pre><code>&lt;?php
      $time = microtime();
      $time = explode(' ', $time);
      $time = $time[1] + $time[0];
      $start = $time;
  
      ini_set("max_execution_time", 300); // 300 seconds = 5 minutes
      ini_set("pcre.backtrack_limit", "100000000"); // default 100k = "100000"
      ini_set("memory_limit", "1024M");
  
      $txt_path = "./test_2.txt";
      $txt_data = @file_get_contents($txt_path) or die("Could not access file: $txt_path");
      //echo $txt_data;
  
      /* BEGIN ARRAY FOR LOOP ENTRIES */
  
      $loop_pattern = "/BEGIN(.*?)END/s";
      preg_match_all($loop_pattern, $txt_data, $matches);
      $loops = $matches[0];
      $loops_count = count($loops);
      //echo("&lt;br&gt;&lt;br&gt;".$loops_count."&lt;br&gt;&lt;br&gt;");
  
      foreach ($loops as $key =&gt; $value) {
          $value = trim($value);
          $pattern = array("/BEGIN(.*?)/", "/END(.*?)/", "/[[:blank:]]+/");
          $replacement = array("", "", " ");
          $value = preg_replace($pattern, $replacement, $value);
          //echo $value."&lt;br&gt;&lt;br&gt;";
  
          preg_match_all( '/^#\d+/m', $value, $matches );
          $keys = $matches[0];
          //print_r($keys);
          //echo "&lt;br&gt;&lt;br&gt;";
  
          $value = preg_replace( '/^#\d+\s*/m', '', $value );
  
          $value = str_replace( "\n", " ", $value );
  
          $pattern = '/'.str_repeat( "('[^']+'|\S+)\s+", count( $keys ) ).'/';
  
          preg_match_all( $pattern, $value, $matches );
          //print_r($matches);
          //echo "&lt;br&gt;&lt;br&gt;";
  
          $loop_dic = array_combine( $keys, array_slice( $matches, 1 ) );
  
          print_r( $loop_dic );
          echo("&lt;br&gt;&lt;br&gt;");
      }
  
      /* END ARRAY FOR LOOP ENTRIES */
  
      /* BEGIN ARRAY FOR NO LOOP ENTRIES */
  
      $txt_data_without_loops = preg_replace( "/BEGIN(.*?)END/s", "", $txt_data );
      //echo $txt_data_without_loops;
  
      $pattern = array("/END(.*?)/", "/[[:blank:]]+/");
      $replacement = array("", " ");
      $txt_data_without_loops_clean = preg_replace($pattern, $replacement, $txt_data_without_loops);
      //echo $txt_data_without_loops_clean;
      preg_match_all( '/^#(.*?)\S+/m', $txt_data_without_loops_clean, $matches );
      $keys = $matches[0];
      //print_r($keys);
      $txt_data_without_loops_clean = preg_replace( '/^#(.*?)\S+\s*/m', '', $txt_data_without_loops_clean );
      //print_r($txt_data_without_loops_clean);
  
      $txt_data_without_loops_clean_no_newline = str_replace( "\n", " ", $txt_data_without_loops_clean );
      //print_r($txt_data_without_loops_clean_no_newline);
      $pattern = '/'.str_repeat( "('[^']+'|\S+)\s+", 1 ).'/';
      preg_match_all( $pattern, $txt_data_without_loops_clean_no_newline, $matches );
      //print_r( $matches[0] );
  
      $no_loop_dic = array_combine( $keys, $matches[0] );
      print_r( $no_loop_dic );
      echo("&lt;br&gt;&lt;br&gt;");
  
      /* END ARRAY FOR NO LOOP ENTRIES */
  
      $time = microtime();
      $time = explode(' ', $time);
      $time = $time[1] + $time[0];
      $finish = $time;
      $total_time = round(($finish - $start), 4);
      echo '&lt;br&gt;&lt;br&gt;&lt;b&gt;Page generated in '.$total_time.' seconds.&lt;/b&gt;&lt;br&gt;&lt;br&gt;';
  ?&gt;
  </code></pre>
  
  <p>As first approach, to obtain the BEGIN-END loops and relative arrays, I read the input file with:</p>
  
  <pre><code>$txt_path = "./input.txt";
  $txt_data = @file_get_contents($txt_path) or die("&lt;b&gt;Could not access file: $txt_path&lt;/b&gt;&lt;br&gt;&lt;br&gt;");
  </code></pre>
  
  <p>that works fine for small files but, with big input files, it generates not-responding times in the browser (I'm testing on Firefox), maybe for a saturation of the RAM to parse the whole big file (my laptop has 3GB of RAM).</p>
  
  <p>I tried the following setting in the php file:</p>
  
  <pre><code>ini_set("max_execution_time", 300); // 300 seconds = 5 minutes
  ini_set("pcre.backtrack_limit", "100000000"); // default 100k = "100000"
  ini_set("memory_limit", "1024M");
  </code></pre>
  
  <p>that seems to solve the problem with some file not so big in size, while, with big files, the process has been finished without errors only without many resources in use in the same moment... So, that isn't the best solution.</p>
  
  <p>Searching on the web, I found <a href="https://www.airpair.com/php/fatal-error-allowed-memory-size" rel="nofollow">this page</a> where I read:</p>
  
  <blockquote>
    <p>If you're reading files, read them line-by-line instead of reading in
    the complete file into memory. Look at <a href="http://php.net/manual/en/function.fgets.php" rel="nofollow">fgets</a> and
    <a href="http://php.net/manual/en/splfileobject.fgets.php" rel="nofollow">SplFileObject::fgets</a>.</p>
  </blockquote>
  
  <p>So I decided to use fgets to read and parse the whole input file.
  After generating an array for all the lines, I need to extract from it each loop, adding it to a loops_array, while I would add the other no_loop key-value couples to another array.</p>
  
  <p>My idea, that seems to be fast, is to find the index of each <code>BEGIN</code>, in this way:</p>
  
  <pre><code>$txt_path = "./test.txt";
  $txt_data = @fopen($txt_path, "rb") or die("&lt;b&gt;Could not access file: $txt_path&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;");
  
  $lines = array();
  while ( !feof($txt_data) ) {
      $line = fgets($txt_data, 1024);
      //echo($line."&lt;br/&gt;&lt;br/&gt;");
      array_push($lines, trim($line));
  }
  
  $lines = array_filter($lines);
  //print_r($lines);
  //echo("&lt;br/&gt;&lt;br/&gt;");
  
  $begins = array_keys($lines, "BEGIN");
  //echo("&lt;b&gt;Begins:&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;");
  //print_r($begins);
  //echo("&lt;br/&gt;&lt;br/&gt;");
  </code></pre>
  
  <p>but now I need to find the index of the first <code>END</code> after each element in the <code>$begins</code> array... If I do:</p>
  
  <pre><code>$ends = array_keys($lines, "END");
  //echo("&lt;b&gt;Ends:&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;");
  //print_r($ends);
  //echo("&lt;br/&gt;&lt;br/&gt;");
  </code></pre>
  
  <p>it considers also the <code>END</code> string in the no_loop zones of the input file, while I should find the index of the first match for the <code>END</code> string, after each <code>BEGIN</code>, combining then them with:</p>
  
  <pre><code>$begins_ends = array_combine($begins, $ends);
  </code></pre>
  
  <p>and extract all the loops with <code>array_slice</code>, adding finally each <code>$loop</code> to a new array, <code>$loops</code>, in a way like this one:</p>
  
  <pre><code>$i = 0;
  $loops = array();
  foreach ($begins_ends as $key =&gt; $value) {
      $begin = trim($key);
      $end = trim($value);
      $loop = array_slice( $lines, $begin, ($end - $begin), false );
      $this_loop = array();
      for ($el=$begin; $el &lt; $end+1; $el++) {
          array_push($this_loop, $lines[$el]);
          unset($lines[$el]);
      }
      array_push($loops, $this_loop);
      $loop = array_values($lines);
      //echo("&lt;b&gt;Loops Dictionary $i:&lt;/b&gt;&lt;br/&gt;&lt;br/&gt;");
      //print_r($loop);
      //echo("&lt;br/&gt;&lt;br/&gt;");
      $i++;
  }
  //print_r($loops);
  //echo("&lt;br/&gt;&lt;br/&gt;");
  </code></pre>
  
  <p>The problem is to obtain the correct <code>$ends</code> array, without considering the <code>END</code> string of the no_loop zones in the input file, obtaining the previous output:</p>
  
  <pre><code>Array ( [#1] =&gt; Array ( [0] =&gt; 1 [1] =&gt; 2 [2] =&gt; 2 [3] =&gt; 2 ) [#2] =&gt; Array ( [0] =&gt; 2015-05-31 [1] =&gt; 2011-04-01 [2] =&gt; 2013-02-24 [3] =&gt; 2014-02-28 ) [#3] =&gt; Array ( [0] =&gt; 2001-11-24 [1] =&gt; ? [2] =&gt; ? [3] =&gt; ? ) [#4] =&gt; Array ( [0] =&gt; 'Name Surname' [1] =&gt; ? [2] =&gt; ? [3] =&gt; 'Name Surname' ) [#5] =&gt; Array ( [0] =&gt; ID_1 [1] =&gt; ID_2 [2] =&gt; ID_3 [3] =&gt; ID_4 ) [#6] =&gt; Array ( [0] =&gt; 0 [1] =&gt; 1 [2] =&gt; 1 [3] =&gt; 2 ) )
  
  Array ( [#7] =&gt; 'value 1' [#8] =&gt; 'value 2' [#9] =&gt; 'value 3' [#10] =&gt; 'value 4' )
  </code></pre>
  
  <p>with the fastest approach and the lowest memory usage, to solve the not-responding times in the browser with files big in size.</p>
  
  <p>Thank you</p>