Why is Julia slow when decoding GZIP and parsing JSON?
Question: The following script reads the gzip file line-by-line and decodes JSON object

    
    
    Pkg.add("GZip")
    Pkg.add("JSON")
    
    
    using GZip
    using JSON
    
    stream = GZip.gzopen(Base.ARGS[1])
    
    _started = time()
    
    i = 0
    
    for line in eachline(stream)
        _j = JSON.parse(line)
    
        if i % 10000 == 0
            println(time()-_started)
        end
    
        i += 1
    
        try
            key_id = _j["some_id"]
        except
            pass
        end
    end
    

output:

    
    
    0.7071459293365479
    20.09155511856079
    37.8870849609375
    

compare to Python:

    
    
    import os
    import sys
    import gzip
    import datetime
    
    import ujson as json
    
    _started = datetime.datetime.now()
    
    fh = gzip.open(sys.argv[1])
    for i, line in enumerate(fh):
    
        if i % 10000 == 0:
            print datetime.datetime.now() - _started
    
        line = json.loads(line)
        try:
            k = line['some_id']
        except:
            pass
    

output:

    
    
    0:00:00.028951
    0:00:01.934277
    0:00:03.821954
    0:00:05.713572
    

Yes, I know, I used ujson which is really fast - but shouldn't Julia be faster
anyway? Instead, it's much slower =(

P.S. Tried many times so it's not related to the OS cache or whatever.

P.P.S. Mac OS 10.7.3, Julia 0.2.1


Answer: Isolating the JSON and GZIP operations is a good suggestion. My guess (only a
guess!) is that most of the time is in JSON parsing; ujson is indeed very
fast. There's no way to guarantee that everything that happens in a language
is fast. PyPy, for example, will generally speed up python code a lot, but in
some cases is still slow, or slower than C extensions.

