title: Memory leak when performing repeated queries using WWW::Mechanize
body: <p>I am trying to find a memory leak in my program. I have found where the leak originates but I cannot fix it.</p>
  <p>The program reads each gene page that is connected to each chromosome as found on Wikipedia <a href="https://en.wikipedia.org/wiki/Category:Genes_by_human_chromosome" rel="nofollow noreferrer"><em>Genes by human chromosome</em></a></p>
  <p>The program extracts the information I am interested in on each gene page, moves onto the next gene page and so on.</p>
  <p>Once it reaches the end of the gene list of the current chromosome, it moves onto the next chromosome until it has gone through each page.</p>
  <p>The program worked on my computer until about 2-3 weeks ago. Since then it started to have this problem.</p>
  <p>I have been monitoring using <code>top</code> and there is a distinct increase in memory usage as the program goes along until it reaches a critical point and my computer crashes.</p>
  <p><strong>As per request I am providing code that can be compiled. I have started it at Chromosome 21 since that one has the fewest number of pages and will therefore take the least amount of time to get through. The memory usage still increases incrementally in this code snippet so hopefully this is enough! Also, the eval statements are in there since querying the wikipedia API sometimes returns nothing instead of the expected JSON. The eval function allowed me to get around this without letting the program die</strong></p>
  <h3>My (Updated) code</h3>
  <pre><code>#!/usr/bin/env perl -w
  
  use common::sense;
  use WWW::Mechanize;
  use URI;
  use HTTP::Request;
  use Cpanel::JSON::XS qw(decode_json);
  
  my ( $self, $registry ) = @_;
  
  my $mech = WWW::Mechanize-&gt;new();
  
  my $root = URI-&gt;new(&quot;http://en.wikipedia.org/w/api.php&quot;);
  
  my $url = $root-&gt;clone();
  
  for my $i ( 21 .. 25 ) {
      my $chrom = $i;
      if ( $chrom == 23 ) {
          $chrom = &quot;M&quot;;
      }
      elsif ( $chrom == 24 ) {
          $chrom = &quot;Y&quot;;
      }
      elsif ( $chrom == 25 ) {
          $chrom = &quot;X&quot;;
      }
      print &quot;Hi!\n The chromosome is $chrom\n&quot;;
  
      my $query = {
          action     =&gt; 'query',
          format     =&gt; 'json',
          list       =&gt; 'categorymembers',
          cmtitle    =&gt; &quot;Category:Genes on human chromosome $chrom&quot;,
          cmlimit    =&gt; 'max',
          cmcontinue =&gt; ''
      };
  
      $url-&gt;query_form($query);
  
      my @gene_pages = ();
      eval {
          while ( my $response = $mech-&gt;get($url) ) {
              my $perl_scalar = decode_json( $response-&gt;decoded_content() )
                  ;    #J Source of malformed JSON string error
              push @gene_pages, @{ $perl_scalar-&gt;{query}-&gt;{categorymembers} };
              my $count = @gene_pages;
  
              # Adapted code to new format for continuing queries
  
              if ( $perl_scalar-&gt;{continue} ) {
                  $query-&gt;{cmcontinue} = $perl_scalar-&gt;{continue}-&gt;{cmcontinue};
                  $url-&gt;query_form($query);
              }
              else {
                  last;
              }
          }
      };
      if ( $@ =~ /malformed/ ) {
          redo;
      }
      my $gene_count = 0;
      eval {
          foreach my $gene_page (@gene_pages) {
              $gene_count++;
              my $url   = $root-&gt;clone();
              my $query = {
                  action  =&gt; 'query',
                  prop    =&gt; 'revisions',
                  format  =&gt; 'json',
                  rvprop  =&gt; 'content|tags|timestamp',
                  pageids =&gt; $gene_page-&gt;{pageid}
              };
              $url-&gt;query_form($query);
  
              #       $log-&gt;debugf(&quot;Requesting: %s&quot;, $url-&gt;as_string());
              my $response    = $mech-&gt;get($url);
              my $content     = $response-&gt;decoded_content();
              my $perl_scalar = decode_json( $response-&gt;decoded_content() )
                  ;    #J Source of malformed JSON string error
              if ( $gene_count % 10 == 0 ) {
                  print &quot;$gene_count gene pages complete\n&quot;;
              }
          }
      };
      print &quot;There were $gene_count genes found for chromosome $chrom\n&quot;;
  
  }
  </code></pre>
  <p>This code has a much larger component but I have excluded it because this is the area that I know has the source of the issue.</p>
  <p>The <code>while</code> loop part that uses <code>WWW::Mechanize</code></p>
  <pre><code>my $response = $mech-&gt;get($url)
  </code></pre>
  <p>is connected to the memory leak.</p>
  <p>If I remove that component and run the program the memory use stays around the same and then adding it back in shows the memory rise incrementally once again.</p>
  <p>Perl version: 5.24.1</p>
  <p>System: Ubuntu 16.04</p>
  <p>Edit: @Borodin Thank you for such a thorough reply! Unfortunately I am still noticing a memory leak on my computer which is making me wonder if there is a larger problem beyond this.</p>
  <p>It still incrementally takes up memory and for now my computer is OK with it but  when I run the full program that includes some web scraping, I don't know that my computer will be able suffice.</p>
  <p>On a potentially related note -- My computer has a weird issue where it sometimes is unable to download files fully (files are truncated despite the download being complete). When I was running your program I got this error a lot:</p>
  <p>**unexpected end of string while parsing JSON string, at character offset 5506 (before &quot;(end of string)&quot;) **</p>
  <p>It seems like it could be related to that issue I am having and I wonder if this contributes to the memory leak problem?</p>