title: Estimating the word count of a file without reading the full file
body: <p>I have a program to process very large files. Now I need to show a progress bar to show the progress of the processing. The program works on a word level, read one line at a time, splitting it into words and processing the words one by one. So while the programs runs, it knows the count of the words processed. If somehow it knows the word count of the file beforehand, it can easily calculate the progress.</p>
  
  <p>The problem is that, the files I am dealing with may be very large and hence it's not a good idea to process the file twice, once to get the total word count and next to run actual processing code.</p>
  
  <p>So I am trying to write a code which can estimate the word count of a file by reading a small portion of it. This is what I have come up with (in Clojure):</p>
  
  <pre><code>(defn estimated-word-count [file]
    (let [^java.io.File file (as-file file)
          ^java.io.Reader rdr (reader file)
          buffer (char-array 1000)
          chars-read (.read rdr buffer 0 1000)]
      (.close rdr)
      (if (= chars-read -1)
        0
        (* 0.001 (.length file) 
          (-&gt; (String. buffer 0 chars-read) tokenize-line count)))))
  </code></pre>
  
  <p>This code reads the first 1000 characters from the file, creates a String from it, tokenizes it to get words, counts the words and then estimates the word count of the file by multiplying it with the length of the file and dividing it by 1000.</p>
  
  <p>When I run this code on a file with English text, I get almost correct word count. But, when I run this on a file with Hindi text (encoded in UTF-8), it return almost double of the real word count.</p>
  
  <p>I understand that this issue is because of the encoding. So is there any way to solve it?</p>
  
  <p><strong>SOLUTION</strong></p>
  
  <p>As <a href="https://stackoverflow.com/questions/3516779/estimating-the-word-count-of-a-file-without-reading-the-full-file/3516914#3516914">suggested by Frank</a>, I determine the byte count of the first 10000 characters and 
  use it to estimate the word count of the file.</p>
  
  <pre><code>(defn chars-per-byte [^String s]
    (/ (count s) ^Integer (count (.getBytes s "UTF-8"))))
  
  (defn estimate-file-word-count [file]
    (let [file (as-file file)
          rdr (reader file)
          buffer (char-array 10000)
          chars-read (.read rdr buffer 0 10000)]
      (.close rdr)
      (if (= chars-read -1)
        0
        (let [s (String. buffer 0 chars-read)]
          (* (/ 1.0 chars-read) (.length file) (chars-per-byte s)
            (-&gt; s tokenize-line count))))))
  </code></pre>
  
  <p>Note that this assume UTF-8 encoding. Also, I decided to read first 10000 chars because it gives a better estimate.</p>