Why can't node handle this regex but python can?
Question: I have a large text file that I am extracting URLs from. If I run:

    
    
    import re
    
    with open ('file.in', 'r') as fh:
       for match in re.findall(r'http://matchthis\.com', fh.read()):
          print match
    

it runs in a second or so user time and gets the URLs I was wanting, but if I
run either of these:

    
    
    var regex = /http:\/\/matchthis\.com/g;
    fs.readFile('file.in', 'ascii', function(err, data) {
       while(match = regex.exec(data))
          console.log(match);
    });
    

OR

    
    
    fs.readFile('file.in', 'ascii', function(err, data) {
       var matches = data.match(/http:\/\/matchthis\.com/g);
       for (var i = 0; i < matches.length; ++i) {
          console.log(matches[i]);
       }
    });
    

I get:

    
    
    FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory
    

What is happening with the `node.js` regex engine? Is there any way I can
modify things such that they work in `node`?

**EDIT:** The error appears to be `fs` centric as this also produces the
error:

    
    
    fs.readFile('file.in', 'ascii', function(err, data) {
    
    });
    

`file.in` is around 800MB.


Answer: You should process the file line by line using the streaming file interface.
Something like this:

    
    
    var fs = require('fs');
    
    var byline = require('byline');
    
    var input = fs.createReadStream('tmp.txt');
    var lines = input.pipe(byline.createStream());
    
    lines.on('readable', function(){
        var line = lines.read().toString('ascii');
    
        var matches = line.match(/http:\/\/matchthis\.com/g);
        for (var i = 0; i < matches.length; ++i) {
            console.log(matches[i]);
        }
    });
    

In this example, I'm using the `byline` module to split the stream into lines
so that you won't miss matches by getting partial chunks of lines per
`.read()` call.

To elaborate more, what you were doing is allocating ~800MB of RAM as a Buffer
(outside of V8's heap) and then converting that to an ASCII string (and thus
transferring it into V8's heap), which will take at least 800MB and likely
more depending on V8's internal optimizations. I believe V8 stores strings as
UCS2 or UTF16, which means each character will be 2 bytes (given ASCII input)
so your string would really be about 1600MB.

Node's max allocated heap space is 1.4GB, so by trying to create such a large
string, you cause V8 to throw an exception.

Python does not have this problem because it does not have a maximum heap size
and will chew through all of your RAM. As others have pointed out, you should
also avoid `fh.read()` in Python since that will copy all the file data into
RAM as a string instead of streaming it line by line with an iterator.

