<p>I'm trying to import an XML file via a web page in a Ruby on Rails application, the code ruby view code is as follows (I've removed HTML layout tags to make reading the code easier)</p>

<pre><code>&lt;% form_for( :fmfile, :url =&gt; '/fmfiles', :html =&gt; { :method =&gt; :post, :name =&gt; 'Form_Import_DDR', :enctype =&gt; 'multipart/form-data' } ) do |f| %&gt;
&lt;%= f.file_field :document, :accept =&gt; 'text/xml', :name =&gt; 'fmfile_document' %&gt;
&lt;%= submit_tag 'Import DDR' %&gt;
&lt;% end %&gt;
</code></pre>

<p>Results in the following HTML form</p>

<pre><code>&lt;form action="/fmfiles" enctype="multipart/form-data" method="post" name="Form_Import_DDR"&gt;&lt;div style="margin:0;padding:0"&gt;&lt;input name="authenticity_token" type="hidden" value="3da97372885564a4587774e7e31aaf77119aec62" /&gt;
&lt;input accept="text/xml" id="fmfile_document" name="fmfile_document" size="30" type="file" /&gt;
&lt;input name="commit" type="submit" value="Import DDR" /&gt;
&lt;/form&gt;
</code></pre>

<p>The Form_Import_DDR method in the 'fmfiles_controller' is the code that does the hard work of reading the XML document in using REXML.  The code is as follows</p>

<pre><code>@fmfile = Fmfile.new
@fmfile.user_id = current_user.id
@fmfile.file_group_id = 1
@fmfile.name = params[:fmfile_document].original_filename

respond_to do |format|
  if @fmfile.save
    require 'rexml/document'
    doc = REXML::Document.new(params[:fmfile_document].read)

    doc.root.elements['File'].elements['BaseTableCatalog'].each_element('BaseTable') do |n|
      @base_table = BaseTable.new
      @base_table.base_table_create(@fmfile.user_id, @fmfile.id, n)
    end
</code></pre>

<p>And it carries on reading all the different XML elements in.</p>

<p>I'm using Rails 2.1.0 and Mongrel 1.1.5 in Development environment on Mac OS X 10.5.4, site DB and browser on same machine.</p>

<p>My question is this.  This whole process works fine when reading an XML document with character encoding UTF-8 but fails when the XML file is UTF-16, does anyone know why this is happening and how it can be stopped?</p>

<p>I have included the error output from the debugger console below, it takes about 5 minutes to get this output and the browser times out before the following output with the 'Failed to open page'</p>

<pre><code>Processing FmfilesController#create (for 127.0.0.1 at 2008-09-15 16:50:56) [POST]
Session ID: BAh7CDoMdXNlcl9pZGkGOgxjc3JmX2lkIiVmM2I3YWU2YWI4ODU2NjI0NDM2
NTFmMDE1OGY1OWQxNSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxh
c2g6OkZsYXNoSGFzaHsABjoKQHVzZWR7AA==--dd9f588a68ed628ab398dd1a967eedcd09e505e0
Parameters: {"commit"=&gt;"Import DDR", "authenticity_token"=&gt;"3da97372885564a4587774e7e31aaf77119aec62", "action"=&gt;"create", "fmfile_document"=&gt;#&lt;File:/var/folders/LU/LU50A0vNHA07S4rxDAOk4E+++TI/-Tmp-/CGI.3001.1&gt;, "controller"=&gt;"fmfiles"}
[4;36;1mUser Load (0.000350)[0m   [0;1mSELECT * FROM "users" WHERE (id = 1) LIMIT 1[0m
[4;35;1mFmfile Create (0.000483)[0m   [0mINSERT INTO "fmfiles" ("name", "file_group_id", "updated_at", "report_created_at", "report_link", "report_version", "option_on_open_account_name", "user_id", "option_default_custom_menu_set", "option_on_close_script", "path", "report_type", "option_on_open_layout", "option_on_open_script", "created_at") VALUES('TheTest_fp7 2.xml', 1, '2008-09-15 15:50:56', NULL, NULL, NULL, NULL, 1, NULL, NULL, NULL, NULL, NULL, NULL, '2008-09-15 15:50:56')[0m

REXML::ParseException (#&lt;Iconv::InvalidCharacter: "਼䙍偒数 (followed by a few thousand similar looking chinese characters)
䙍偒数潲琾", ["\n"]&gt;
/Library/Ruby/Site/1.8/rexml/encodings/ICONV.rb:7:in `conv'
/Library/Ruby/Site/1.8/rexml/encodings/ICONV.rb:7:in `decode'
/Library/Ruby/Site/1.8/rexml/source.rb:50:in `encoding='
/Library/Ruby/Site/1.8/rexml/parsers/baseparser.rb:210:in `pull'
/Library/Ruby/Site/1.8/rexml/parsers/treeparser.rb:21:in `parse'
/Library/Ruby/Site/1.8/rexml/document.rb:190:in `build'
/Library/Ruby/Site/1.8/rexml/document.rb:45:in `initialize'
</code></pre>

[{'answer_id': 64068, 'author': 'tomafro', 'author_id': 7126, 'author_profile': 'https://Stackoverflow.com/users/7126', 'pm_score': 1, 'selected': False, 'text': '<p>Rather than a rails/mongrel problem, it sounds more likely that there\'s an issue either with your XML file or with the way REXML handles it.  You can check this by writing a short script to read your XML file directly (rather than within a request) and seeing if it still fails.</p>\n\n<p>Assuming it does, there are a couple of things I\'d look at.  First, I\'d check you are running the latest version of REXML.  A couple of years ago there was a bug (<a href="http://www.germane-software.com/projects/rexml/ticket/63" rel="nofollow noreferrer">http://www.germane-software.com/projects/rexml/ticket/63</a>) in its UTF-16 handling.  </p>\n\n<p>The second thing I\'d check is if you\'re issue is similar to this: <a href="http://groups.google.com/group/rubyonrails-talk/browse_thread/thread/ba7b0585c7a6330d" rel="nofollow noreferrer">http://groups.google.com/group/rubyonrails-talk/browse_thread/thread/ba7b0585c7a6330d</a>.  If so you can try the workaround in that thread.</p>\n\n<p>If none of the above helps, then please reply with more information, such as the exception you are getting when you try and read the file.</p>\n'}, {'answer_id': 65207, 'author': 'Matt Haughton', 'author_id': 6106, 'author_profile': 'https://Stackoverflow.com/users/6106', 'pm_score': 0, 'selected': False, 'text': '<p>Since getting this to work requires me to only change the encoding attribute of the first XML element to have the value UTF-8 instead of UTF-16, the XML file is actually UTF-8 and labelled wrongly by the application that generates it.</p>\n\n<p>The XML file is a FileMaker DDR export produced by FileMaker Pro Advanced 8.5 on OS X 10.5.4</p>\n'}, {'answer_id': 262462, 'author': 'Dema', 'author_id': 407003, 'author_profile': 'https://Stackoverflow.com/users/407003', 'pm_score': 0, 'selected': False, 'text': "<p>Have you tried doing this using JRuby? I've heard Unicode strings are better supported in JRuby.</p>\n\n<p>One other thing you can try is to use another XML parsing library, such as libxml ou Hpricot. </p>\n\n<p>REXML is one of the slowest Ruby XML libraries you can use and might not scale.</p>\n"}, {'answer_id': 291489, 'author': 'George Stocker', 'author_id': 16587, 'author_profile': 'https://Stackoverflow.com/users/16587', 'pm_score': 0, 'selected': False, 'text': '<p>Actually, I think your problem may be related to the problem I just detailed in <a href="https://stackoverflow.com/questions/291455/xml-data-at-root-level-is-invalid">this post</a>.  If I were you, I\'d open it up in TextPad in Binary mode and see if there are any Byte Order Marks before your XML starts.</p>\n'}]
['https://Stackoverflow.com/questions/63632', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/6106/']