title: Should the file opening interface in a C++ library use UTF-8 on Windows?
body: <p>I'm working on a library (pugixml) that, among other things, provides file load/save API for XML documents using narrow-character C strings:</p>
  
  <pre><code>bool load_file(const char* path);
  bool save_file(const char* path);
  </code></pre>
  
  <p>Currently the path is passed verbatim to <code>fopen</code>, which means that on Linux/OSX you can pass a UTF-8 string to open the file (or any other byte sequence that is a valid path), but on Windows you have to use Windows ANSI encoding - UTF-8 won't work.</p>
  
  <p>The document data is (by default) represented using UTF-8, so if you had an XML document with a file path, you would not be able to pass the path retrieved from the document to <code>load_file</code> function as is - or rather, this would not work on Windows. The library provides alternative functions that use <code>wchar_t</code>:</p>
  
  <pre><code>bool load_file(const wchar_t* path);
  </code></pre>
  
  <p>But using them requires extra effort for encoding UTF8 to wchar_t.</p>
  
  <p>A different approach (that is used by SQlite and GDAL - not sure if there are other C/C++ libraries that do that) involves treating the path as UTF-8 on Windows (which would be implemented by converting it to UTF-16 and using a <code>wchar_t</code>-aware function like <code>_wfopen</code> to open the file).</p>
  
  <p>There are different pros and cons that I can see and I'm not sure which tradeoff is best.</p>
  
  <p>On one hand, using a consistent encoding on all platforms is definitely good. This would mean that you can use file paths extracted from the XML document to open other XML documents. Also if the application that uses the library adopts UTF-8 it does not have to do extra conversions when opening XML files through the library.</p>
  
  <p>On the other hand, this means that behavior of file loading is no longer the same as that of standard functions - so file access through the library is not equivalent to file access through standard <code>fopen</code>/<code>std::fstream</code>. It seems that while some libraries take the UTF-8 path, this is largely an unpopular choice (is this true?), so given an application that uses many third-party libraries, it may increase confusion instead of helping developers.</p>
  
  <p>For example, passing <code>argv[1]</code> into <code>load_file</code> currently works for paths encoded using system locale encoding on Windows (e.g. if you have a Russian locale you can load any files with Russian names like that, but you won't be able to load files with Japanese characters). Switching to UTF-8 will mean that only ASCII paths work unless you retrieve the command-line arguments in some other Windows-specific way.</p>
  
  <p>And of course this would be a breaking change for some users of the library.</p>
  
  <p>Am I missing any important points here? Are there other libraries that take the same approach? What is better for C++ - being consistently inconsistent in file access, or striving for uniform cross-platform behavior?</p>
  
  <p>Note that the question is about the default way to open the files - of course nothing prevents me from adding <em>another</em> pair of functions with _utf8 suffix or indicating the path encoding in some other way.</p>