text: Chances are, some of you have run into the issue with the
  invalid byte sequence
  in UTF-8 error when dealing with user-submitted data. A Google
  search shows
  that my hunch isn’t off.
  Among the search results are plenty of answers—some using the deprecated iconv library—that might lead you to a sufficient fix. However, among the slew of queries are few answers on how to reliably replicate and test the issue.
  In developing the Griddler gem we ran into some cases where the data being posted back to our controller had invalid UTF-8 bytes. For Griddler, our failing case needs to simulate the body of an email having an invalid byte, and encoded as UTF-8.
  What are valid and invalid bytes? This table on Wikipedia tells us bytes 192, 193, and 245-255 are off limits. In ruby’s string literal we can represent this by escaping one of those numbers:
  > "hi \255" => "hi \xAD" real-world, enterprise-level, business-critical test case:
  invalid_byte_spec.rb
  require 'rspec' def replace_name(body, name) body.gsub(/joel/, name) end describe 'replace_name' do it 'removes my name' do body = "hello joel" replace_name(body, 'hank').should eq "hello hank" end it 'clears out invalid UTF-8 bytes' do body = "hello joel\255" replace_name(body, 'hank').should eq "hello hank" end end
  The first test passes as expected, and the second will fail as expected but not with the error we want. By adding that extra byte we should see an exception raised similar to what we simulated in IRB. Instead it’s failing in the comparison with the expected value.
  1) replace_name clears out invalid UTF-8 bytes Failure/Error: replace_name(body, 'hank').should eq "hello hank" expected: "hello hank" got: "hello hank\xAD" (compared using ==) # ./invalid_byte_spec.rb:17:in `block (2 levels) in <top (required)>'
  Why isn’t it failing properly? If we pry into our running test we find out that
  inside our file the strings being passed around are encoded as
  ASCII-8BIT
  instead of
  UTF-8.
  [2] pry(#<RSpec::Core::ExampleGroup::Nested_1>)> body.encoding => #<Encoding:ASCII-8BIT>
  As a result we’ll have to force that string’s encoding to UTF-8:
  it 'clears out invalid UTF-8 bytes' do body = "hello joel\255".force_encoding('UTF-8') replace_name(body, 'hank').should_not raise_error(ArgumentError) replace_name(body, 'hank').should eq "hello hank" end
  By running the test now we will see our desired exception
  1) replace_name clears out invalid UTF-8 bytes Failure/Error: body.gsub(/joel/, name) ArgumentError: invalid byte sequence in UTF-8 # ./invalid_byte_spec.rb:4:in `gsub' # ./invalid_byte_spec.rb:4:in `replace_name' # ./invalid_byte_spec.rb:17:in `block (2 levels) in <top (required)>' Finished in 0.00426 seconds 2 examples, 1 failure
  Now that we’re comfortably in the red part of red/green/refactor we
  can move on to getting this passing by updating our
  replace_name method.
  def replace_name(body, name) body .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '') .gsub(/joel/, name) end
  And the test?
  Finished in 0.04252 seconds 2 examples, 0 failures
  For such a small piece of code we admittedly had to jump through some hoops. Through that process, however, we learned a bit about character encoding and how to put ourselves in the right position—through the red/green/refactor cycle—to fix bugs we will undoubtedly run into while writing software.