title: Manipulating big array in a more memory-efficient way
body: <p>I'm currently writing an API for an app that interfaces with a large database and need to retrieve large amounts of data from it, then spit it out as JSON.</p>
  
  <p>I'm using CodeIgniter (CI) as a database interface but I don't think it's relevant here. I'm running into memory limits and I have unfortunately no way of increasing the limit because the shared hosting service won't do it.</p>
  
  <p>I get about 56k rows from the database, which get put into an array by CI (zero-indexed, pretty standard). Each row has 7 fields.</p>
  
  <p>All is well until I start looping through the array to modify data. The script runs into a memory limit error after a few loop iterations even though I'm just modifying the original array, and not allocating new variables, I think.</p>
  
  <pre><code>Allowed memory size of 134217728 bytes exhausted
  </code></pre>
  
  <p>Below is the code I'm using:</p>
  
  <pre><code>$query = $this-&gt;db-&gt;get('table');
  if ($query-&gt;num_rows() &gt; 0) {
      $result = $query-&gt;result_array();
      foreach ($result as $k =&gt; $v) {
          foreach($v as $key =&gt; $value) {
              if ($key === 'column_name') {
                  $result[$k][$key] = json_decode($value);
                  continue;
              }
              if ($value == null) {
                  $result[$k][$key] = '';
              } else if (ctype_digit($value)) {
                  $result[$k][$key] = (int) $result[$k][$key];
              }
          }
      }
      return $result;
  }
  </code></pre>
  
  <p>Just decoding some json and casting to integers or empty strings, nothing fancy. But I will get memory limit errors on any line that mutates the <code>$result</code> array. Even if I remove the (memory-intensive) <code>json_decode</code> I will still get an error on line that simply casts to an <code>int</code>.</p>
  
  <p>What's more, even if I remove the whole <code>foreach</code>, I get a memory limit error later on when I use <code>json_encode</code> to generate the API response.</p>
  
  <p>I'm totally lost, and I really need this amount of data to be output all at once, no idea how to make this more memory-efficient (maybe with like buffers or something? never dived into this).</p>
  
  <p><strong>EDIT</strong>: for anyone interested, I managed to cut down on memory usage somewhat by making an unbuffered query to the database. This way, only 1 copy of the data is stored in the array. I also removed the foreach and treat each field specifically. The main problem, however, is probably how <a href="https://stackoverflow.com/a/38558486/3446439">PHP stores arrays</a>. Here's the new code:</p>
  
  <pre><code>$query = $this-&gt;db-&gt;get('table');
  $result = [];
  while ($row = $query-&gt;unbuffered_row('array')) {
      if ($row['column1'] == '[]') {
          $row['column1'] = [];
      } else {
          $row['column1'] = json_decode($row['column1']);
      }
      $row['column2'] = (int) $row['column2'];
      $row['column3'] = (int) $row['column3'];
      $row['column4'] = is_null($row['column4']) ? '' : (int) $row['column4'];
      $row['column5'] = is_null($row['column5']) ? '' : (int) $row['column5'];
  
      $result[] = $row;
  }
  
  return $result;
  </code></pre>