title: Clojure : Get "OutOfMemoryError Java heap space" when parsing big log file
body: <p>all.<br/>
  I want to parse big log files using Clojure.<br/>
  And the structure of each line record is "UserID,Lantitude,Lontitude,Timestamp".<br/>
  My implemented steps are:<br/>
  ----> Read log file &amp; Get top-n user list <br/>
  ----> Find each top-n user's records and store in separate log file (UserID.log) .</p>
  
  <p>The implement source code :</p>
  
  <pre><code>;======================================================
  (defn parse-file
    ""
    [file n]
    (with-open [rdr (io/reader file)]
      (println "001 begin with open ")
      (let [lines (line-seq rdr)
            res (parse-recur lines)
            sorted
            (into (sorted-map-by (fn [key1 key2]
                                   (compare [(get res key2) key2]
                                            [(get res key1) key1])))
                  res)]
        (println "Statistic result : " res)
        (println "Top-N User List : " sorted)
        (find-write-recur lines sorted n)
        )))
  
  (defn parse-recur
    ""
    [lines]
    (loop [ls  lines
           res {}]
      (if ls
        (recur (next ls)
                 (update-res res (first ls))) 
        res)))
  
  (defn update-res
    ""
    [res line]
    (let [params (string/split line #",")
          id     (if (&gt; (count params) 1) (params 0) "0")]
      (if (res id)
        (update-in res [id] inc)
        (assoc res id 1))))
  
  (defn find-write-recur
    "Get each users' records and store into separate log file"
    [lines sorted n]
    (loop [x n
           sd sorted
           id (first (keys sd))]
      (if (and (&gt; x 0) sd)
        (do (create-write-file id
                               (find-recur lines id))
            (recur (dec x)
                   (rest sd)
                   (nth (keys sd) 1))))))
  
  (defn find-recur
    ""
    [lines id]
    (loop [ls lines
             res []]
      (if ls
        (recur (next ls)
                 (update-vec res id (first ls)))
        res)))
  
  (defn update-vec
    ""
    [res id line]
    (let [params (string/split line #",")
          id_        (if (&gt; (count params) 1) (params 0) "0")]
          (if (= id id_ )
            (conj res line)
            res)))
  
  (defn create-write-file
    "Create a new file and write information into the file."
    ([file info-lines]
     (with-open [wr (io/writer (str MAIN-PATH file))]
       (doseq [line info-lines] (.write wr (str line "\n")))
       ))
    ([file info-lines append?]
     (with-open [wr (io/writer (str MAIN-PATH file) :append append?)]
       (doseq [line info-lines] (.write wr (str line "\n"))))
     ))
  ;======================================================
  </code></pre>
  
  <p>I tested this clj in REPL with command (parse-file "./DATA/log.log" 3), and get the results:<br/></p>
  
  <p>Records-----Size-----Time----Result <br/>
  1,000-------42KB-----&lt;1s-----OK <br/>
  10,000------420KB----&lt;1s-----OK<br/>
  100,000-----4.3MB----3s------OK<br/>
  1,000,000---43MB-----15s-----OK<br/>
  6,000,000---258MB---->20M----"OutOfMemoryError Java heap space  java.lang.String.substring (String.java:1913)"<br/></p>
  
  <p>======================================================<br/>
  Here is the question: <br/>
  1. how can i fix the error when i try to parse big log file , like > 200MB<br/>
  2. how can i optimize the function to run faster ?<br/>
  3. there are logs more than 1G size , how can the function deal with it.<br/></p>
  
  <p>I am still new to Clojure, any suggestion or solution will be appreciate~<br/>
  Thanks<br/></p>