title: Why is Ruby Base64 encoding conversion losing data and how I can route around?
body: <p>I am spinning up a a Rails UI that talks to a Grape API. This is the second instance of this program. The first instance works well. The second instance's Grape API, however, appears to be corrupting data before sending it over the wire.</p>
  
  <p>I need the image to go from file > json > http > db. Right now I am doing that by sending the file like so: file > string > encode to url-safe base64 > to_json > http > decode > save to sqlite3 db with ActiveRecord. I'm led to believe the image data is corrupted by my converting it to base64 based on the below. However, since the Grape is all JSON, the characters must be encoded before sending (since, at least as far as Ruby's JSON library is concerned, invalid UTF-8 == invalid JSON).</p>
  
  <p>So I either have to know:</p>
  
  <ol>
  <li>How to allow Grape API to send non-JSON (raw file string) or</li>
  <li>How to decode the string and avoid the error message</li>
  </ol>
  
  <hr>
  
  <p>Opening a file and converting its contents to url-safe Base64.</p>
  
  <pre><code>File.open("#{folder}/#{file_name}", "rb:UTF-8") do |image|
    file_as_string = image.read
  end
   =&gt; "iVBORw0K ... # truncated for length
  </code></pre>
  
  <p>Things go weird right away. IRB does the expected - encodes as UTF-8.</p>
  
  <pre><code>file_as_string.encoding.name
   =&gt; "UTF-8"
  </code></pre>
  
  <p>BUT. The server logs <code>ASCII-8BIT</code>. I cannot explain this. Every file is topped with Ruby's magic UTF-8 comment. Linux <code>$LANG</code> is set to <code>en_US.UTF-8</code>.</p>
  
  <p>OK, but when Base64 converts I lose the plot anyway. Even in IRB, starting with UTF-8, it down coverts. Why US-ASCII? Regardless, why is compatibility is lost?.</p>
  
  <pre><code>Base64.urlsafe_encode64(file_as_string).encoding.name
   =&gt; "US-ASCII"
  Base64.urlsafe_decode64(Base64.urlsafe_encode64(file_as_string)).encoding.name
   =&gt; "ASCII-8BIT"
  Base64.urlsafe_decode64(Base64.urlsafe_encode64(file_as_string)).encode("UTF-8")
  Encoding::UndefinedConversionError: "\x89" from ASCII-8BIT to UTF-8
      from (irb):27:in `encode'
      from (irb):27
      from /home/me/.rvm/rubies/ruby-2.2.1/bin/irb:11:in `&lt;main&gt;'
  </code></pre>
  
  <p>Note that the error here in IRB is the same as if I a) don't base64 encode the string before Grape tries to_json and b) when I try to decode and call <code>.save</code> the string to a model attribute on the Rails side.</p>
  
  <p>The file itself is binary (if that matters?)</p>
  
  <pre><code>$ file -bi /path/to/file.png
  image/png; charset=binary
  </code></pre>
  
  <hr>
  
  <p>Solutions I've tried, or am unwilling to try:</p>
  
  <p><strong>Sending over the raw image.read</strong></p>
  
  <p>This is a JSON API, so Grape converts to JSON before sending the data over the wire -- meaning any response must be valid JSON, as far as I understand it. If I try to send the raw string over, the automatically-called <code>.to_json</code> throws the same error.</p>
  
  <p><strong>Force-encoding the results</strong></p>
  
  <p>The output is not a readable png.</p>
  
  <p><strong>Downgrading</strong></p>
  
  <p>The original instance is Ruby 1.9.2 and CentOS 6.3. The new instance is Ruby 2.2.1 and CentOS 7. I'm generally committed to moving forward, so I'd rather develop some solution, even if not backward compatible, then rollback Ruby and my OS.</p>
  
  <p><strong>Not using UTF-8</strong></p>
  
  <p>Rails's <code>config/application.rb</code> has the line <code>config.encoding = "utf-8"</code> and <code>config/environment.rb</code> has the lines <code>Encoding.default_external = Encoding::UTF_8; Encoding.default_internal = Encoding::UTF_8</code> I hope not to have to give up UTF-8 compatibility just for this one issue.</p>
  
  <hr>
  
  <p>So is there a way to serve a file directly in Grape, bypassing the <code>to_json</code> call? Or is there a different encoding safe for JSON-ing and sending over http?</p>