title: Unreliable/Flakey Capybara/AngularJS Integration Tests With Timing Issues
body: <h1>How do I make these tests reliably pass?</h1>
  
  <p>Currently these tests are flakey.<br>
  Sometimes they pass. Sometimes they fail.<br>
  Below is the setup, code and output demonstrating this issue.<br>
  Suggestions to overcome this issue will be greatly appreciated and I am sure will help many others, so please comment!</p>
  
  <h3>Test Code Environment</h3>
  
  <ol>
  <li>Rails 3.2</li>
  <li>RSpec 2.x</li>
  <li>Capybara  </li>
  <li>Poltergeist  </li>
  <li>PhantomJS</li>
  <li>AngularJS</li>
  <li>Google Chrome Version 47.0.2526.106 (64-bit)</li>
  </ol>
  
  <h3>Testing Gems from Gemfile.lock</h3>
  
  <pre><code>capybara (2.1.0)
  database_cleaner (0.7.1)
  debug_inspector (0.0.2)
  guard-bundler (0.1.3)
  guard-livereload (1.2.0)
  guard-rspec (2.1.2)
  jasminerice (0.0.10)
  pg (0.17.1)
  phantomjs (2.1.1.0)
  poltergeist (1.4.1)
  protractor-rails (0.0.17)
  pry (0.9.12)
  rack (1.4.7)
  rack-test (0.6.3)
  rails (3.2.21)
  rails-assets-angular (1.3.20)
  rspec-rails (2.11.4)
  simplecov (0.8.2)
  sprockets (2.2.3)
  zeus (0.13.3)
  zeus-parallel_tests (0.2.1)
  </code></pre>
  
  <h3>Things I have tried</h3>
  
  <ol>
  <li>Ensure that I use Capybara's waiting DSL matchers  </li>
  <li>Ensure that my database cleaner is correctly setup  </li>
  <li>Test every page item assuming that it might not be on the page and could still be loading  </li>
  <li>Narrow down inconsistent tests  </li>
  <li>Run inconsistent test alone  </li>
  <li><p>Identify code Capybara DSLs that are the trigger to inconsistent test results.  </p>
  
  <ul>
  <li>i.e. creating a new record and assuming the page has redirected and that the record is on the page click_on  </li>
  </ul>
  
  <p>or  </p>
  
  <ul>
  <li>.click not consistently 'working'   </li>
  </ul></li>
  <li>Upgrade Capybara to the latest version (in a seperate branch)  </li>
  <li>Upgraded Poltergeist and RSpec to the latest version (in a seperate branch, still working on this)</li>
  </ol>
  
  <h2>Resources I used</h2>
  
  <p>[1] <a href="https://github.com/jnicklas/capybara#the-dsl" rel="nofollow">Capybara The DSL</a><br>
  [2] <a href="http://www.railsonmaui.com/tips/rails/capybara-phantomjs-poltergeist-rspec-rails-tips.html" rel="nofollow">Capybara, PhantomJs, Poltergeist, and Rspec Tips</a><br>
  And many more...</p>
  
  <h3>How tests were run</h3>
  
  <p><code>rspec spec/integration/costings/show_costing_spec.rb --format documentation</code></p>
  
  <h3>Test Code</h3>
  
  show_costing_spec.rb
  
  <pre><code>require "spec_helper"
  
  RSpec.describe "Show a new costing in the listing," do
  
    before :each do
      admin_sign_in
      create_costing("test1")
    end
  
    it "shows the costing after creation" do
      within "#costings_table" do
        expect(page).to have_css("#name", text: "test1")
      end
    end
  
    it "shows the details of the new costing after creation" do
      expect(page).to have_content("Costings")
      within "#costings_table" do
        expect(page).to have_content("test1")
        all("#show").last.click
      end
  
      expect(page).to have_content("Costing Details")
      expect(page).to have_css("#name", text: "test1")
    end
  end  
  </code></pre>
  
  spec_helper.rb
  
  <pre><code># This file is copied to spec/ when you run 'rails generate r spec:install'  
  ENV["RAILS_ENV"] ||= 'test'
  require File.expand_path("../../config/environment", __FILE__)
  # Add library functions here so we can test them.
  require File.expand_path(File.dirname(__FILE__) + "/../lib/general")
  require 'rspec/rails'
  require 'rspec/autorun'
  
  # Integration Testing
  require 'capybara/poltergeist'
  Capybara.register_driver :poltergeist_debug do |app|
   Capybara::Poltergeist::Driver.new(app, :inspector =&gt; true)  
  end
  Capybara.javascript_driver = :poltergeist_debug
  Capybara.default_driver = :poltergeist_debug
  
  # Capybara Integration Test Helpers
  def admin_sign_in
    visit "/login"
    #Create staff member in database
    Staff.make!(:admin)
    #Log In
    fill_in "staff_username", with: "adminstaff"
    fill_in "staff_password", with: "password"
    click_button "login"
  end
  
  def create_costing(item)
    visit "/api#/costings"
    click_on "new_btn"
    within "#form_costing" do
      find("#name", match: :first).set("#{item}")
      find("#description", match: :first).set("test description")    
      find("#from_date", match: :first).set("15/02/2016")
      find("#cost_hourly_cents", match: :first).set("1.00")
      click_on "create_btn"
    end
  end
  
  RSpec.configure do |config|
    config.before(:suite) do
      # Requires supporting ruby files with custom matchers and macros, etc,
      # in spec/support/ and its subdirectories.
      require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
      Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
    end
  
    # Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
    config.fixture_path = "#{::Rails.root}/spec/fixtures"
  
    # Allow a 'focus' tag so that we can run just a few tests which we are currently working on
    config.treat_symbols_as_metadata_keys_with_true_values = true
    config.filter_run focus: true
    config.run_all_when_everything_filtered = true
    config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]
  
    # Defer Garbage Collection
    config.before(:all) { DeferredGarbageCollection.start }
    config.after(:all)  { DeferredGarbageCollection.reconsider }
  
    # If you're not using ActiveRecord, or you'd prefer not to run each of your
    # examples within a transaction, remove the following line or assign false
    # instead of true.
    config.use_transactional_fixtures = false
    # config.infer_spec_type_from_file_location!
  
    # Configure Database Cleaner
    config.include Capybara::DSL
    config.before(:suite) do
      DatabaseCleaner.clean_with(:truncation)
    end
  
    config.before(:each) do
      DatabaseCleaner.strategy = :transaction
    end
  
    config.before(:each, :js =&gt; true) do
      DatabaseCleaner.strategy = :truncation
    end
  
    config.before(:each) do
      DatabaseCleaner.start
    end
  
    config.after(:each) do
      DatabaseCleaner.clean
    end
  end
  </code></pre>
  
  <h3>Test Results</h3>
  
  <p><code>Test Run 1: Failing</code>  </p>
  
  <blockquote>
    <p>Run options:
    include {:focus=>true}
    exclude {:slow=>true}</p>
    
    <p>All examples were filtered out; ignoring {:focus=>true}</p>
    
    <p>Show a new costing in the listing,
      shows the costing after creation
      shows the details of the new costing after creation (FAILED - 1)</p>
    
    <p>Failures:</p>
    
    <p>1) Show a new costing in the listing,<br>
       shows the details of the new costing after creation<br>
         Failure/Error: expect(page).to have_content("test1")<br>
           expected #has_content?("test1") to return true, got false<br>
         # ./spec/integration/costings/show_costing_spec.rb:20:in block (3 levels) in <br>
         # ./spec/integration/costings/show_costing_spec.rb:19:in block (2 levels) in   </p>
    
    <p>Finished in 5.46 seconds
    2 examples, 1 failure</p>
  </blockquote>
  
  <p><code>Test Run 2: Passing</code></p>
  
  <blockquote>
    <p>Run options:
       include {:focus=>true}
       exclude {:slow=>true}</p>
    
    <p>All examples were filtered out; ignoring {:focus=>true}</p>
    
    <p>Show a new costing in the listing,<br>
      shows the costing after creation<br>
      shows the details of the new costing after creation</p>
    
    <p>Finished in 3.57 seconds
     2 examples, 0 failures</p>
  </blockquote>
  
  <h3>Update 1</h3>
  
  <p>Upgraded testing gems to the following versions:<br>
      <strong>capybara            (2.6.2)   from (2.1.0)</strong><br>
      <strong>database_cleaner    (1.5.1)   from (0.7.1)</strong><br>
      debug_inspector     (0.0.2)<br>
      guard-bundler       (0.1.3)<br>
      guard-livereload    (1.2.0)<br>
      guard-spec          (2.1.2)<br>
      jasminerice         (0.0.10)<br>
      pg                  (0.17.1)<br>
      phantomjs           (2.1.1.0)<br>
      <strong>poltergeist         (1.9.0)   from (1.4.1)</strong><br>
      protractor-rails    (0.0.17)<br>
      <strong>pry                 (0.10.3)  from (0.9.12)</strong><br>
      rack                (1.4.7)<br>
      rack-test           (0.6.3)<br>
      rails               (3.2.21)<br>
      <strong>rails-assets-angular (1.4.9)   from (1.3.20)</strong><br>
      <strong>rspec-rails         (3.4.2)   from (2.11.4)</strong><br>
      simplecov           (0.8.2)<br>
      sprockets           (2.2.3)<br>
      zeus                (0.13.3)<br>
      zeus-parallel_tests (0.2.1)  </p>
  
  <p><code>Result 1</code></p>
  
  <blockquote>
    <p>Unfortunately upgrading these gems did not seem to make a difference and my tests were still flakey. </p>
  </blockquote>
  
  <h3>Update 2</h3>
  
  <p>I implemented Tom Walpole's suggestions. Ensured that my admin_sign_in waits for sign_in to complete.</p>
  
  <p>Also updated my database_cleaner setup as Tom suggested.</p>
  
  <p><code>Result 2</code></p>
  
  <blockquote>
    <p>For my stack these changes did not seem to have an effect.  </p>
    
    <p><strong>Note:</strong> If one is not using AngularJS I feel these changes would have made a difference. So thank you Tom for your suggestions.</p>
  </blockquote>
  
  <h3>Update 3</h3>
  
  <p>I needed to get more information on what was happening during my test runs. There are suggestions on the net to log, use screen shot saving gems and the like but I did not feel like these would be the most time efficient. I wanted to specify where I wanted the test to pause at and view the contents of variables and form fields. In a browser would be ideal. </p>
  
  <p><strong>What I Used</strong><br>
  I was using "save_and_open_page" and "print page.html" to debug.  </p>
  
  <p><strong>What I Moved To</strong><br>
  As I was running RSpec 3.4.2 I added a debug helper method:  </p>
  
  <p>rails_helper.rb</p>
  
  <pre><code>def debugit
    puts current_url
    require 'pry'
    binding.pry
  end
  </code></pre>
  
  <p><code>Result 3</code></p>
  
  <blockquote>
    <p>A URL would be printed in the console and the test would be paused. At this stage I would be able to navigate to the URL, login, navigate to the test page and view what the Capybara test had done. </p>
    
    <p>This enabled me to identify that the source of my problems arose when the test was using capybara's fill_in DSL.
     In some test runs the fields would be populated correctly and the form would be submitted. 
    In the other scenario the form would be filled in correctly but the submit button would be hit too quickly. The result here is that a record was created but input fields of name and description were not persisted. </p>
  </blockquote>
  
  <h3>Update 4</h3>
  
  <p>I discovered that because I was using AngularJS input forms and tables, AngularJS required a tiny bit of time to bind to the input fields. If it was not allowed this time the input data would not be saved. </p>
  
  <p>Capybara provides waiting methods such as "within" and "find". I used these but they did not help with the AngularJS binding time issue. 
  I found ng-if could be used to create an if statement to wait for particular item that would signify the AngularJS bindings to the form fields complete.</p>
  
  <p>So I used Capybara waiting methods to wait for the fields I wanted to fill and I used AngularJS' ng-if not to show the fields until they are ready.</p>
  
  <p><strong>Implementation</strong><br>
  index.html.erb </p>
  
  <pre><code>&lt;div  ng-if="tableParams.data"&gt;
    &lt;table id="costings_table ng-table="tableParams" class="table"&gt;
      &lt;td id="field1"&gt;{{table.field1}}&lt;/td&gt;
      &lt;td id="field2"&gt;{{table.field2}}&lt;/td&gt;
    &lt;/table&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p><code>Result 4</code>  </p>
  
  <blockquote>
    <p>Tests finally pass! However I have all these find methods with xpath ensuring specific and difficult to target items are waited on...</p>
  </blockquote>
  
  <h3>Update 5</h3>
  
  <p>Even though in my gemfile I was running the gem phantomJS version 2.1.1 my command line version was only 1.X. This proved to be significant. </p>
  
  <p>I updated my command line phantomJS version to 2.1.1. At the same time I ensured all my input boxes, buttons, tables, headings had unique ids. I then was able to remove all find(:xpath) occurances without breaking the tests. </p>
  
  <p><code>Result 5</code>  </p>
  
  <blockquote>
    <p>This suite of tests now reliably pass all the time! Exactly what I wanted! Yes!</p>
  </blockquote>