title: Bash equivalent MD5 function from Ruby
body: <p>I'm using the <a href="https://github.com/mgomes/api_auth" rel="nofollow">ApiAuth gem</a> to authenticate API requests. Now I need to write a shell script that uses cURL to send test requests. So I need to generate an MD5 of the POST body and base64 encode it so that it matches what ApiAuth does on the server:</p>
  
  <p>My shell script:</p>
  
  <pre><code>query="{\"document\":{\"recipient_id\":\"$ACCESS_ID\",\"data\":{\"id\":\"$ACCESS_ID\"}},\"vendor_string\":\"test\",\"patient\":{\"document\":{\"recipient_id\":\"$ACCESS_ID\",\"data\":{\"id\":\"$ACCESS_ID\"}}}}"
  
  # need to figure how to get a base64 encoded md5 the same way Ruby does
  content_md5=$(echo -n "$query" | openssl md5 -binary | base64)
  content_type='application/json'
  request_uri="$API_BASE/test"
  httpdate=$(date -u +"%a, %_d %b %Y %H:%M:%S GMT")
  accept_header='application/vnd.test+json; version=1'
  
  canonical_string="$content_type,$content_md5,$request_uri,$httpdate"
  signature=$(echo -n "$canonical_string" | openssl dgst -sha1 -hmac "$SECRET_KEY" -binary | base64)
  
  curl -H "Authorization: APIAuth $ACCESS_ID:$signature"\
       -H "Content-MD5: $content_md5" \
       -H "Date: $httpdate" \
       -H "Accept: $accept_header" \
       -H "Content-type: $content_type" \
       -d $query \
       -v \
       $request_uri
  </code></pre>
  
  <p>The first thing that fails is comparing the Content-MD5 that I send with the content MD5 that ApiAuth calculates:</p>
  
  <p><a href="https://github.com/mgomes/api_auth/blob/master/lib/api_auth/base.rb#L37" rel="nofollow">https://github.com/mgomes/api_auth/blob/master/lib/api_auth/base.rb#L37</a></p>
  
  <pre><code>def authentic?(request, secret_key)
    return false if secret_key.nil?
  
    return !md5_mismatch?(request) &amp;&amp; signatures_match?(request, secret_key) &amp;&amp; !request_too_old?(request)
  end
  </code></pre>
  
  <p>Here the <a href="https://github.com/mgomes/api_auth/blob/master/lib/api_auth/base.rb#L71" rel="nofollow">md5_mismatch?(request)</a> method returns false. it uses these methods to calculate the MD5:</p>
  
  <p><a href="https://github.com/mgomes/api_auth/blob/master/lib/api_auth/request_drivers/action_controller.rb" rel="nofollow">https://github.com/mgomes/api_auth/blob/master/lib/api_auth/request_drivers/action_controller.rb</a></p>
  
  <pre><code>def calculated_md5
    if @request.env.has_key?('RAW_POST_DATA')
      body = @request.raw_post
    else
      body = ''
    end
    md5_base64digest(body)
  end
  </code></pre>
  
  <p><a href="https://github.com/mgomes/api_auth/blob/master/lib/api_auth/helpers.rb" rel="nofollow">https://github.com/mgomes/api_auth/blob/master/lib/api_auth/helpers.rb</a></p>
  
  <pre><code>def b64_encode(string)
    if Base64.respond_to?(:strict_encode64)
      Base64.strict_encode64(string)
    else
      # Fall back to stripping out newlines on Ruby 1.8.
      Base64.encode64(string).gsub(/\n/, '')
    end
  end
  
  def md5_base64digest(string)
    if Digest::MD5.respond_to?(:base64digest)
      Digest::MD5.base64digest(string)
    else
      b64_encode(Digest::MD5.digest(string))
    end
  end
  </code></pre>
  
  <p>So I'm thinking it boils down to matching exactly what's going on with:</p>
  
  <pre><code>Digest::MD5.base64digest
  </code></pre>
  
  <p>My attempt was:</p>
  
  <pre><code>content_md5=$(echo -n "$query" | openssl md5 -binary | base64)
  </code></pre>
  
  <p><strong>How can I make the bash script equivalent to the ruby method?</strong></p>
  
  <p>I've tried with and without the <code>-binary</code> flag.</p>
  
  <p>I've checked that the <code>$query</code> in bash is the exact same as <code>@request.raw_post</code> in Ruby and there's no trailing newlines since I'm using <code>echo -n</code>.</p>
  
  <p><strong>Update:</strong></p>
  
  <p>Output from bash:</p>
  
  <pre><code>echo $query
  {"document":{"recipient_id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o","data":{"id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o"}},"vendor_string":"kipusystems","patient":{"document":{"recipient_id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o","data":{"id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o"}}}}
  
  echo $content_md5
  Lsb/vxJKHUxyRAqMhOMeOw==
  </code></pre>
  
  <p>Output from ruby:</p>
  
  <pre><code>puts body
  {"document":{"recipient_id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o","data":{"id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o"}},"vendor_string":"kipusystems","patient":{"document":{"recipient_id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o","data":{"id":"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o"}}}}
  
  puts md5_base64digest(body)
  /DdffT+N+sZZjaTC5TJNcg==
  </code></pre>
  
  <p>I selected and copied the <code>$query</code> and <code>body</code> strings out of the terminals that ran the bash script and rails server respectively. They're both exactly the same in that sense, how can I further narrow down this problem?</p>
  
  <p><strong>Update 2:</strong> Maybe some character encoding issue?</p>
  
  <p>I pasted this literal text into the (mac bash) shell prompt:</p>
  
  <pre><code>echo -n "{\"document\":{\"recipient_id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\",\"data\":{\"id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\"}},\"vendor_string\":\"kipusystems\",\"patient\":{\"document\":{\"recipient_id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\",\"data\":{\"id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\"}}}}" | openssl dgst -md5 -binary | base64
  </code></pre>
  
  <p>And that outputs: <code>/DdffT+N+sZZjaTC5TJNcg==</code> which is good! That's what the Ruby side outputs. Ok cool.</p>
  
  <p>But when I run my shell script with that exact literal command I just pasted above, it outputs: <code>Lsb/vxJKHUxyRAqMhOMeOw==</code> which is the same as the content-md5 I originally started with (script originally posted). </p>
  
  <p>When I run <code>echo $LANG</code> I get <code>en_US.UTF-8</code>.</p>
  
  <p><strong>Update 3:</strong></p>
  
  <p>I run the shell script with:</p>
  
  <pre><code>sh script.sh
  </code></pre>
  
  <p>And that outputs <code>Lsb/vxJKHUxyRAqMhOMeOw==</code> when I echo out this command:</p>
  
  <pre><code>echo -n "{\"document\":{\"recipient_id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\",\"data\":{\"id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\"}},\"vendor_string\":\"kipusystems\",\"patient\":{\"document\":{\"recipient_id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\",\"data\":{\"id\":\"lwzvZixLvVLL50qasfoO2YvMz9UzNlxg8HBOEj8NV_o\"}}}}" | openssl dgst -md5 -binary | base64
  </code></pre>
  
  <p><strong>Update 4:</strong></p>
  
  <p>Weird! So, I've been running (the above posted shell script) using <code>sh script.sh</code> and that has shown me an md5 result that was different from what I was seeing in Ruby. Now, I <code>chmod +x</code>'ed the script and ran it directly: <code>script.sh</code> and now I get the correct md5!!</p>
  
  <p>But, now the <code>signatures_match?</code> method in ApiAuth returns false still :'(</p>