title: Fast reading of unicode files in windows in R
body: <p>I always use <code>fread</code> from <strong>data.table</strong> package to read in large tables. But apparently it does not support reading unicode files in Windows (Windows 7 Professional to be more precise)</p>
  
  <p>Here is the file I tried to read:</p>
  
  <pre><code>A,B
  ą,ž
  ū,į
  ų,ė
  š,ę
  </code></pre>
  
  <p>It works fine if I read it in Mac OS X, or I read it with <code>read.csv</code> with option <code>encoding=UTF-8</code>. Unfortunately <code>fread</code> does not have this <a href="https://github.com/Rdatatable/data.table/issues/563" rel="nofollow">option</a>. </p>
  
  <p>So are there other fast ways to read unicode tables in Windows, or I should just use other OS? Or am I missing something obvious?</p>
  
  <p>Here is the output of <code>sessionInfo():</code></p>
  
  <pre><code>R version 3.1.3 (2015-03-09)
  Platform: x86_64-w64-mingw32/x64 (64-bit)
  Running under: Windows 7 x64 (build 7601) Service Pack 1
  
  locale:
  [1] LC_COLLATE=English_United States.1252 
  [2] LC_CTYPE=English_United States.1252   
  [3] LC_MONETARY=English_United States.1252
  [4] LC_NUMERIC=C                          
  [5] LC_TIME=English_United States.1252    
  
  attached base packages:
  [1] stats     graphics  grDevices utils     datasets  methods   base     
  
  other attached packages:
  [1] data.table_1.9.4
  
  loaded via a namespace (and not attached):
  [1] chron_2.3-45   plyr_1.8.1     Rcpp_0.11.5    reshape2_1.4.1 stringr_0.6.2 
  </code></pre>
  
  <p><strong>Update:</strong> Pasting the output as requested.</p>
  
  <pre><code>&gt; aa&lt;-fread("F:/R/unicode_test2.csv",verbose=TRUE)
  
  Input contains no \n. Taking this to be a filename to open
  File opened, filesize is 0.000000 GB.
  Memory mapping ... ok
  Detected eol as \r\n (CRLF) in that order, the Windows standard.
  Positioned on line 1 after skip or autostart
  This line is the autostart and not blank so searching up for the last non-blank ... line 1
  Detecting sep ... ','
  Detected 2 columns. Longest stretch was from line 1 to line 5
  Starting data input on line 1 (either column names or first row of data). First 10 characters: Ä„,B
  All the fields on line 1 are character fields. Treating as the column names.
  Count of eol: 5 (including 1 at the end)
  Count of sep: 4
  nrow = MIN( nsep [4] / ncol [2] -1, neol [5] - nblank [1] ) = 4
  Type codes (   first 5 rows): 44
  Type codes: 44 (after applying colClasses and integer64)
  Type codes: 44 (after applying drop or select (if supplied)
  Allocating 2 column slots (2 - 0 dropped)
  Read 4 rows. Exactly what was estimated and allocated up front
     0.000s (  0%) Memory map (rerun may be quicker)
     0.000s (  0%) sep and header detection
     0.000s (  0%) Count rows (wc -l)
     0.000s (  0%) Column type detection (first, middle and last 5 rows)
     0.000s (  0%) Allocation of 4x2 result (xMB) in RAM
     0.000s (  0%) Reading data
     0.000s (  0%) Allocation for type bumps (if any), including gc time if triggered
     0.000s (  0%) Coercing data already read in type bumps (if any)
     0.000s (  0%) Changing na.strings to NA
     0.001s        Total
  
  &gt; aa
     Ä„  B
  1: Ä… Å¾
  2: Å« Ä¯
  3: Å³ Ä—
  4: Å¡ Ä™
  &gt; aa$A
  [1] "Ä…" "Å«" "Å³" "Å¡"
  &gt; aa$B
  [1] "Å¾" "Ä¯" "Ä—" "Ä™"
  
  &gt; bb &lt;- read.csv("F:/R/unicode_test.csv",encoding="UTF-8",strings=FALSE)
  &gt; bb
    A B
  1 a ž
  2 u i
  3 u e
  4 š e
  &gt; bb$B
  [1] "ž" "į" "ė" "ę"
  &gt; bb$A
  [1] "ą" "ū" "ų" "š"
  </code></pre>