title: Rails Rspec Suite Has Failures But They Pass When Run Individually
body: <p>Facts:</p>
  
  <ol>
  <li>Running my entire suite of specs will result in 21 consistent errors out of 610 specs.  </li>
  <li>If I run any individual spec file (for instance: messages_controller_spec.rb), they will all pass.  </li>
  <li>If I run any of the failed specs individually, they will each pass.</li>
  </ol>
  
  <p>These errors are mostly ActionMailer failures, but some are otherwise. 
  One confusing aspect is that some of the specs fail because there was an extra row in the database than expected, while others fail because there was one less row than expected. That is, if it were a cleaning or cache issue, it seems like it should be consistently one more or one less. </p>
  
  <p>I’m currently running Rails 4.1.1, Ruby 2.0.0p451, Rspec 2.14.8, Sidekiq via inline!</p>
  
  <p><strong>Gemfile(for test)</strong></p>
  
  <pre><code>group :development, :test do
    gem 'better_errors'
    gem 'binding_of_caller'
    gem 'faker'
    gem 'guard-rspec'
    gem 'pry'
    gem 'rspec-rails'
    gem 'spork-rails'
    gem 'sqlite3'
    gem 'thin'
  end
  
  group :test do
    gem 'capybara'
    gem 'capybara-email'
    gem 'capybara-webkit'
    gem 'database_cleaner'
    gem 'fabrication'
    gem 'launchy'
    gem 'selenium-webdriver'
    gem 'shoulda-matchers'
    gem 'webmock'
    gem 'vcr'
  end
  </code></pre>
  
  <p>Please note, I’ve checked out about a dozen similar questions that did not work to fix this. So, to clarify: </p>
  
  <ol>
  <li>I’m not using ARGV</li>
  <li>I’m not using before(:all) - I use before(:each). </li>
  <li>I have attempted Rails.cache.clear before both the :suite &amp; the :each spec.</li>
  <li>I have database cleaner set to :truncation for all cleaning (slower, but better results than :transaction - have had issues with reloading updated values using :transaction)</li>
  </ol>
  
  <p>For convenience sake, maybe some examples will help:</p>
  
  <p><strong>scheduler_spec.rb (showing test that fails when whole suite is run)</strong></p>
  
  <pre><code>require 'spec_helper'
  require 'rake'
  require 'sidekiq/testing'
  Sidekiq::Testing.inline!
  
  describe "scheduler", :vcr do
    describe ":wipe_abandoned_images" do
      let!(:abandoned_old_image)   { Fabricate(:image) }
      let!(:abandoned_young_image) { Fabricate(:image) }
      let!(:adopted_image)         { Fabricate(:image) }
      let(:run_cleaner) do
        Rake::Task[:wipe_abandoned_images].reenable
        Rake.application.invoke_task :wipe_abandoned_images
      end
  
    before do
        abandoned_young_image.update_columns(listing_id: nil, updated_at: 6.days.ago)
        abandoned_old_image.update_columns(  listing_id: nil, updated_at: 9.days.ago)
        Rake.application.rake_require 'tasks/scheduler'
        Rake::Task.define_task(:environment) #Stub env. Rspec runs the App, so dont want Rake to run it again.
    end
  
    context "for claimed images" do
        it "leaves the image" do
          adopted_image_id = adopted_image.id
          run_cleaner
            expect(Image.all.count                ).to eq(2)
            expect(Image.find(adopted_image_id) ).to be_present
        end
      end
    end
  end
  </code></pre>
  
  <p>Note that I'm using Sidekiq's inline! testing configuration on prior apps with good success &amp; without this issue. </p>
  
  <p><strong>forgot_passwords_controller_spec.rb (showing test that fails when whole suite is run)</strong></p>
  
  <pre><code>require 'spec_helper'
  require 'sidekiq/testing'
  Sidekiq::Testing.inline!
  
  describe ForgotPasswordsController do
    let!(:jen) { Fabricate(:user, email: 'jen@example.com') }
  
    describe "POST create" do
      context "with valid email provided" do
        before { post :create, email: 'jen@example.com' }
        after do
          ActionMailer::Base.deliveries.clear
          Sidekiq::Worker.clear_all
        end
  
        it 'sends the reset email to the users provided email' do
          expect(ActionMailer::Base.deliveries.count).to eq(1)
        end
      end
    end
  end
  </code></pre>
  
  <h1>Here's what happens when I run the specs in various ways:</h1>
  
  <p><strong>ForgotPasswordsController Specs pass via RubyTest plugin in SublimeText2</strong></p>
  
  <blockquote>
    <p>2014-08-28T03:42:43Z 32968 TID-ov5g65p44 INFO: Sidekiq client with
    redis options {} ........... Finished in 0.95479 seconds 11 examples,
    0 failures Randomized with seed 40226 [Finished in 5.8s]</p>
  </blockquote>
  
  <p><strong>Scheduler Tests pass via RubyTest in SublimeText2</strong></p>
  
  <blockquote>
    <p>.Rationing out invitations to users... done. .Rationing out
    invitations to users... done. ..Sweeping the server for abandoned
    images... 2014-08-28T01:49:02Z 32426 TID-owjt9ggh8 INFO: Sidekiq
    client with redis options {} done. .Sweeping the server for abandoned
    images... done. .Sweeping the server for abandoned images... done.
    .</p>
    
    <p>Finished in 1.52 seconds 7 examples, 0 failures Randomized with seed
    37996 [Finished in 8.6s]</p>
  </blockquote>
  
  <p><strong>Tests pass via rspec in Console</strong>
  $ rspec ./spec/lib/tasks/scheduler_spec.rb</p>
  
  <blockquote>
    <p>.Rationing out invitations to users... done. .Rationing out
    invitations to users... done. ..Sweeping the server for abandoned
    images... 2014-08-28T02:14:43Z 32456 TID-ouiui9g8c INFO: Sidekiq
    client with redis options {} done. .Sweeping the server for abandoned
    images... done. .Sweeping the server for abandoned images... done.
    .</p>
    
    <p>Finished in 1.32 seconds 7 examples, 0 failures Randomized with seed
    19172</p>
  </blockquote>
  
  <p><strong>Tests fail when run entire suite of Rspec</strong>
  <em>These failures will pass if run individually or as the spec file they are from.</em>
  $ rspec</p>
  
  <blockquote>
    <p>Finished in 49.71 seconds 610 examples, 21 failures, 10 pending</p>
  </blockquote>
  
  <p>Failed examples:</p>
  
  <p><strong><em>Sometimes it has an extra value</em></strong></p>
  
  <blockquote>
    <p>13) scheduler :wipe_abandoned_images for abandoned images under 1
    week old leaves the image
         Failure/Error: expect(Image.all.count      ).to eq(2)</p>
  
  <pre><code>   expected: 2
          got: 3
  
     (compared using ==)
   # ./spec/lib/tasks/scheduler_spec.rb:78:in `block (4 levels) in &lt;top (required)&gt;'
  </code></pre>
  </blockquote>
  
  <p><strong><em>Sometimes it loses or doesn’t load a value</em></strong> </p>
  
  <blockquote>
    <p>18) ForgotPasswordsController POST create with valid email provided
    sends the reset email to the users provided email
         Failure/Error: expect(ActionMailer::Base.deliveries.count).to eq(1)</p>
  
  <pre><code>   expected: 1
          got: 0
  
     (compared using ==)
   # ./spec/controllers/forgot_passwords_controller_spec.rb:22:in `block (4 levels) in &lt;top (required)&gt;'
  </code></pre>
  </blockquote>
  
  <p><strong>Here's a list of the failures</strong></p>
  
  <pre><code>rspec ./spec/controllers/messages_controller_spec.rb:161 # MessagesController POST create message about listing to user from guest with valid information with EXISTING, UN-confirmed guest with EXPIRED token sends another confirmation email with link to the guest
  rspec ./spec/controllers/messages_controller_spec.rb:114 # MessagesController POST create message about listing to user from guest with valid information with NEW, UN-confirmed, and valid guest email sends an invitation for the guest to be put on safe-email list
  rspec ./spec/controllers/invitations_controller_spec.rb:30 # InvitationsController POST create with valid email &amp; available invitations sends an email
  rspec ./spec/controllers/invitations_controller_spec.rb:33 # InvitationsController POST create with valid email &amp; available invitations sends an email to the recipient_email address
  rspec ./spec/controllers/users_controller_spec.rb:161 # UsersController POST create with invitation token in params with valid token &amp; input confirmation email sending sends the email to the registering user
  rspec ./spec/controllers/users_controller_spec.rb:158 # UsersController POST create with invitation token in params with valid token &amp; input confirmation email sending sends the email
  rspec ./spec/controllers/users_controller_spec.rb:164 # UsersController POST create with invitation token in params with valid token &amp; input confirmation email sending sends an email with a confirmation link in the body
  rspec ./spec/controllers/users_controller_spec.rb:354 # UsersController GET confirm_with_token with valid token has a welcome message in the email
  rspec ./spec/controllers/users_controller_spec.rb:348 # UsersController GET confirm_with_token with valid token sends a welcome email
  rspec ./spec/controllers/users_controller_spec.rb:351 # UsersController GET confirm_with_token with valid token sends the welcome email to the user
  rspec ./spec/controllers/searches_controller_spec.rb:19 # SearchesController GET search GET search with specific category selected returns the matching OR partial-matching table row objects
  rspec ./spec/controllers/searches_controller_spec.rb:22 # SearchesController GET search GET search with specific category selected only returns values from the selected category
  rspec ./spec/lib/tasks/scheduler_spec.rb:75 # scheduler :wipe_abandoned_images for abandoned images under 1 week old leaves the image
  rspec ./spec/lib/tasks/scheduler_spec.rb:68 # scheduler :wipe_abandoned_images for abandoned images over 1 week old deletes the images
  rspec ./spec/lib/tasks/scheduler_spec.rb:84 # scheduler :wipe_abandoned_images for claimed images leaves the image
  rspec ./spec/controllers/forgot_passwords_controller_spec.rb:24 # ForgotPasswordsController POST create with valid email provided sets the email subject to notify the user of the reset link
  rspec ./spec/controllers/forgot_passwords_controller_spec.rb:27 # ForgotPasswordsController POST create with valid email provided sends the link with token in the body of the email
  rspec ./spec/controllers/forgot_passwords_controller_spec.rb:21 # ForgotPasswordsController POST create with valid email provided sends the reset email to the users provided email
  rspec ./spec/controllers/reset_passwords_controller_spec.rb:70 # ResetPasswordsController POST create with a valid token sets the email subject to notify the user of the reset password
  rspec ./spec/controllers/reset_passwords_controller_spec.rb:67 # ResetPasswordsController POST create with a valid token sends a confirmation email to the user that their password has been changed
  rspec ./spec/controllers/reset_passwords_controller_spec.rb:73 # ResetPasswordsController POST create with a valid token sends the link with token in the body of the email
  </code></pre>