title: AWS S3 file uploads returning 403:Forbidden with Carrierwave/fog gems in Rails
body: <p>I've been scratching my head over this for several days. Currently working through <a href="https://www.railstutorial.org/book/user_microposts" rel="nofollow noreferrer">Chapter 13 (user microposts) of the Rails Tutorial</a>, and while my app works fine in development, I cannot seem to get the image uploads to AWS S3 running in production (details <a href="https://www.railstutorial.org/book/user_microposts#sec-image_upload_in_production" rel="nofollow noreferrer">here in the tutorial</a>). The app utililzes ASW S3 buckets for storage, and <a href="https://github.com/carrierwaveuploader/carrierwave#using-amazon-s3" rel="nofollow noreferrer">CarrierWave</a>/Fog gems for file upload.</p>
  
  <p>When I open my production heroku app, everything works fine except the image upload. When I try to upload an image on a new micropost, I get a generic 'We're sorry, but something went wrong.' error in the browser. Heroku logs show a :status 403 error "Forbidden" when trying to reach the bucket (detailed logs below). </p>
  
  <p>Others have seemed to have similar problems. In most cases it seems the solution is to <a href="https://stackoverflow.com/questions/30350785/trying-to-set-up-amazons-s3-bucket-403-forbidden-error-setting-permissions">set proper IAM user permissions</a> or <a href="https://stackoverflow.com/questions/26910251/carrierwave-s3-403-forbidden-error">set the S3 bucket policy</a>, but I am fairly confident I have those set up correctly for two reasons:</p>
  
  <ol>
  <li>I am able to use the web GUI on that user account to upload and manage files in the bucket, so I know it has access.</li>
  <li>I installed the aws command line tool, configured it with the same user access key and secret key used in my web app, and I <strong>am</strong> able to upload and retrieve information from the bucket through the CLI. </li>
  </ol>
  
  <p>Still I have tried several bucket policies and user permissions, including the Amazon S3 full access permission (I believe this is the most general), and several more specific versions (see latest below). </p>
  
  <p>Other things I have tried that seem to work for others: </p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/30358220/writing-an-iam-policy-and-cors-configuration-for-amazon-s3">Updating CORS configuration for the bucket</a> (although honestly I do not fully understand CORS or if it is necessary for accessing buckets. Again I started with a general config, not a specific one, and still can't get it to work). </li>
  <li><a href="https://stackoverflow.com/questions/16677579/carrierwave-upload-with-amazon-s3-403-forbidden-error">Using different gems for storage</a>, such as <a href="https://www.rubydoc.info/gems/carrierwave-aws/0.7.1" rel="nofollow noreferrer">carrierwave-aws</a> in lieu of fog.</li>
  <li>Setting fog's <code>config.delete_tmp_file_after_storage = false</code> -- two examples of success for this: <a href="https://github.com/carrierwaveuploader/carrierwave/issues/220#issuecomment-791313" rel="nofollow noreferrer">1</a> | <a href="https://github.com/carrierwaveuploader/carrierwave/issues/1662" rel="nofollow noreferrer">2</a></li>
  </ul>
  
  <p>Being a new developer, I feel I don't have the technical sophistication yet to diagnose this one, and frankly it is a bit discouraging, so I am hoping for help. Here are some questions I have and possible lines of inquiry I would like to look further into:</p>
  
  <ul>
  <li>Question: If I am able to access the bucket with the aws-cli, shouldn't I also be able to use heroku with carrierwave/fog using the same credentials, or am I misunderstanding something about how heroku reaches for the S3 bucket?  </li>
  <li>Possible issue: Carrierwave or ImageMagick creates a temp file when the image is uploaded. In some cases it seems that may interfere with the bucket upload. Someone gave a <a href="https://stackoverflow.com/a/28895566/10881795">vague answer about this here</a>, but I do not understand what action might help get rid of that problem. </li>
  <li>Possible issue: Someone seemed to have issues with Rails <a href="https://github.com/rails/strong_parameters/issues/114" rel="nofollow noreferrer">strong params</a> while doing this. I could not quite understand how to debug this... When I use byebug to open an interactive console in the server at the error location (in my the micropost_controller #create method), right after the file upload, the image is in the params hash, but the <code>:picture</code> key of the instance variable <code>@micropost</code> is nil (at the same time, the <code>:content</code> key is not nil, it contains whatever text I submitted with the picture, as it should).</li>
  </ul>
  
  <p><a href="https://github.com/bchalman/sample_app" rel="nofollow noreferrer">Link to the rest of the code on GitHub</a> if it helps.</p>
  
  <p>Sorry for the long-windedness. Any guidance would be greatly appreciated. </p>
  
  <hr>
  
  <p>Errors:</p>
  
  <pre><code>2019-01-07T08:11:37.684069+00:00 app[web.1]: F, [2019-01-07T08:11:37.683926 #22] FATAL -- : [363c5115-f872-43b4-857a-10d1d7d11737] Excon::Error::Forbidden (Expected(200) &lt;=&gt; Actual(403 Forbidden)
  2019-01-07T08:11:37.684074+00:00 app[web.1]: excon.error.response
  2019-01-07T08:11:37.684077+00:00 app[web.1]: :body          =&gt; "&lt;?xml version=\"1.0\" encoding=\"UTF-8\"?&gt;\n&lt;Error&gt;&lt;Code&gt;AccessDenied&lt;/Code&gt;&lt;Message&gt;Access Denied&lt;/Message&gt;&lt;RequestId&gt;***&lt;/RequestId&gt;&lt;HostId&gt;***=&lt;/HostId&gt;&lt;/Error&gt;"
  2019-01-07T08:11:37.684079+00:00 app[web.1]: :cookies       =&gt; [
  2019-01-07T08:11:37.684081+00:00 app[web.1]: ]
  2019-01-07T08:11:37.684082+00:00 app[web.1]: :headers       =&gt; {
  2019-01-07T08:11:37.684085+00:00 app[web.1]: "Connection"       =&gt; "close"
  2019-01-07T08:11:37.684086+00:00 app[web.1]: "Content-Type"     =&gt; "application/xml"
  2019-01-07T08:11:37.684088+00:00 app[web.1]: "Date"             =&gt; "Mon, 07 Jan 2019 08:11:36 GMT"
  2019-01-07T08:11:37.684090+00:00 app[web.1]: "Server"           =&gt; "AmazonS3"
  2019-01-07T08:11:37.684092+00:00 app[web.1]: "x-amz-id-2"       =&gt; "***"
  2019-01-07T08:11:37.684094+00:00 app[web.1]: "x-amz-request-id" =&gt; "***"
  2019-01-07T08:11:37.684096+00:00 app[web.1]: }
  2019-01-07T08:11:37.684098+00:00 app[web.1]: :host          =&gt; "bucket-name.s3-us-west-1.amazonaws.com"
  2019-01-07T08:11:37.684099+00:00 app[web.1]: :local_address =&gt; "*********"
  2019-01-07T08:11:37.684101+00:00 app[web.1]: :local_port    =&gt; ******
  2019-01-07T08:11:37.684103+00:00 app[web.1]: :path          =&gt; "/uploads/micropost/picture/306/ocean2.jpeg"
  2019-01-07T08:11:37.684104+00:00 app[web.1]: :port          =&gt; 443
  2019-01-07T08:11:37.684106+00:00 app[web.1]: :reason_phrase =&gt; "Forbidden"
  2019-01-07T08:11:37.684108+00:00 app[web.1]: :remote_ip     =&gt; "*******"
  2019-01-07T08:11:37.684110+00:00 app[web.1]: :status        =&gt; 403
  2019-01-07T08:11:37.684111+00:00 app[web.1]: :status_line   =&gt; "HTTP/1.1 403 Forbidden\r\n"
  2019-01-07T08:11:37.684113+00:00 app[web.1]: ):
  2019-01-07T08:11:37.684217+00:00 app[web.1]: F, [2019-01-07T08:11:37.684147 #22] FATAL -- : [363c5115-f872-43b4-857a-10d1d7d11737]
  2019-01-07T08:11:37.684392+00:00 app[web.1]: F, [2019-01-07T08:11:37.684328 #22] FATAL -- : [363c5115-f872-43b4-857a-10d1d7d11737] app/controllers/microposts_controller.rb:7:in `create'
  </code></pre>
  
  <hr>
  
  <p>Bucket policy:</p>
  
  <pre><code>{
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::***********:user/user-name"
              },
              "Action": [
                  "s3:AbortMultipartUpload",
                  "s3:ListBucket",
                  "s3:GetObject",
                  "s3:PutObject",
                  "s3:PutObjectAcl",
                  "s3:DeleteObject",
                  "s3:GetObjectVersion"
              ],
              "Resource": [
                  "arn:aws:s3:::bucket-name/*",
                  "arn:aws:s3:::bucket-name"
              ]
          },
          {
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::*******:user/user-name"
              },
              "Action": [
                  "s3:ListBucket",
                  "s3:GetBucketLocation"
              ],
              "Resource": "arn:aws:s3:::bucket-name",
              "Condition": {}
          }
      ]
  }
  </code></pre>
  
  <hr>
  
  <p>CORS:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"&gt;
  &lt;CORSRule&gt;
      &lt;AllowedOrigin&gt;*&lt;/AllowedOrigin&gt;
      &lt;AllowedMethod&gt;GET&lt;/AllowedMethod&gt;
      &lt;AllowedMethod&gt;POST&lt;/AllowedMethod&gt;
      &lt;AllowedMethod&gt;PUT&lt;/AllowedMethod&gt;
      &lt;MaxAgeSeconds&gt;3000&lt;/MaxAgeSeconds&gt;
      &lt;AllowedHeader&gt;*&lt;/AllowedHeader&gt;
  &lt;/CORSRule&gt;
  &lt;CORSRule&gt;
      &lt;AllowedOrigin&gt;*&lt;/AllowedOrigin&gt;
      &lt;AllowedMethod&gt;GET&lt;/AllowedMethod&gt;
      &lt;AllowedMethod&gt;POST&lt;/AllowedMethod&gt;
      &lt;AllowedMethod&gt;PUT&lt;/AllowedMethod&gt;
      &lt;MaxAgeSeconds&gt;3000&lt;/MaxAgeSeconds&gt;
      &lt;AllowedHeader&gt;*&lt;/AllowedHeader&gt;
  &lt;/CORSRule&gt;
  &lt;/CORSConfiguration&gt;
  </code></pre>
  
  <hr>
  
  <p>Current Initializer: <strong>carrier_wave.rb</strong></p>
  
  <pre><code>if Rails.env.production?
    CarrierWave.configure do |config|
      config.fog_credentials = {
        # Configuration for Amazon S3
        :provider              =&gt; 'AWS',
        :aws_access_key_id     =&gt; ENV['S3_ACCESS_KEY'],
        :aws_secret_access_key =&gt; ENV['S3_SECRET_KEY'],
        :region                =&gt; ENV['S3_REGION']
      }
      config.fog_directory     =  ENV['S3_BUCKET']
    end
  end
  </code></pre>
  
  <hr>
  
  <p>picture_uploader.rb</p>
  
  <pre><code>class PictureUploader &lt; CarrierWave::Uploader::Base
    include CarrierWave::MiniMagick
    process resize_to_limit: [400, 400]
  
    if Rails.env.production?
      storage :fog
    else
      storage :file
    end
  
    # Override the directory where uploaded files will be stored.
    # This is a sensible default for uploaders that are meant to be mounted:
    def store_dir
      "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
    end
  
    # Add a white list of extensions which are allowed to be uploaded.
    def extension_whitelist
      %w(jpg jpeg gif png)
    end
  end
  </code></pre>