Problem with js: true and Rspec/Capybara/PhantomJS/Poltergiest

Testing
1

I’m developing on a Nitrous box so to run any feature tests involving JS I use PhantomJS installed as one of their ‘Autoparts’. I’m using the Poltergeist gem, installed and configured as per the gem’s docs.

I’m getting a consistent failure behavior where the test scenario after one with js: true fails with an error like this:

Failure/Error: click_button 'Sign in'
RuntimeError:                                                                                                                                              
       Called id for nil, # etc

If I write a simple test:

feature 'User signs in' do
  scenario 'with valid details' do
    user = create(:user)
    visit sign_in_path
    fill_in 'Email', with: @user.email
    fill_in 'Password', with: 'password'
    click_button 'Sign in'
    expect(page).to have_content('Welcome')
  end
end

then duplicate the scenario a few times in the spec file and change one of them to read:

scenario 'with valid details WITH JS', js: true do

then I get a consistent results:

with valid details
with valid details WITH JS
with valid details (FAILED - 1)

It’s always the test after the JS test that fails. I’ve looked through the test.log but it all looks fine until the 500 internal server error line then it just stops.

I’ve read a lot about shared database connections but when I tried adding the lines suggested here: http://blog.plataformatec.com.br/2011/12/three-tips-to-improve-the-performance-of-your-test-suite/ I get the same results with or without it.

I’m using Database Cleaner with the standard setup suggested by Avdi Grimm (who advises against the use of the ‘shared db connection’ technique).

The fact that it’s the test after the JS test that always fails seems to point to a problem with cleaning the database but all my other 200+ tests run fine. The only problems I have are with feature tests that require JS.

Any pointers would be most welcome.

3

This is a multi-tenant app and it actually seems to be an issue with creating and referencing a tenant.

When I changed the before(:all) to before(:each) in the following it all started to work fine:

RSpec.configure do |config|
  config.before(:all) do
    @tenant = Tenant.where(name: 'Tenant').first_or_create!
  end
end

I can see why I need to build a new tenant before each test now that I’m using Database Cleaner but I’m not sure why it only became a problem when I was running js: true tests.

Anyway, seems to be fixed now.

4

If you’re using Avdi’s database cleaner setup, then the database will be cleared completely between JavaScript specs. This means that any data creating by a before(:all) block will be deleted before the specs run. If you create data using before(:each), it will be created after the database is cleared.

The reason it worked before JavaScript got involved is that non-JavaScript tests use the transaction strategy. In that strategy, a database transaction is started before each test and rolled back at the end. This means that data which existed in the database before the test won’t be cleared, and will remain after the transaction is rolled back.

5

Thanks @jferris for the thorough explanation. Makes total sense now, I was just a little confused what the difference between :all and :each was.

These are the dangers of copying/pasting code without knowing exactly what it does. It may solve the immediate problem but when it becomes the root of a later problem you have no idea why.

Thanks again.

6

I’m having issues with correctly setting up my DB cleaner config for a multitenant rails app where JS tests are involved. Most recently we discovered that we need to add PgTools.set_search_path "test" for our “test” Tenant . If we don’t add that, the seeds.rb seemed to populate data in the public schema.

But I’m still not able to get my spec completely passing and I’m trying to figure out why. @weavermedia Would it be possible for you to share your complete DB cleaner setup? Below is our current setup that we’ve come up with as part of the Rspec configure block in spec_helper.rb(This is a Rails 4.0.13 app with RSpec version is 2.99 so we aren’t yet using rails_helper.rb). Also the seed related setting are taken from this link.

I’m having a hard time debugging this, I see some debugging tips mentioned here(I’m yet to fully try them) but if there’s something specific that really helps you do effectively debugging for such cases, kindly let me know.

The immediate below two settings wrt Capybara is because I was facing some DNS issue and I wasn’t able to fill in details in a modal . More details here

Capybara.app_host = "http://test.lvh.me"
Capybara.always_include_port = true


RSpec.configure do |config|
  config.add_setting(:seed_tables)
  config.seed_tables = %w(table_name_1)
  config.use_transactional_fixtures = false

  config.before(:suite) do
    PgTools.set_search_path "test"
    DatabaseCleaner.clean_with(:truncation, except: config.seed_tables)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
    Tenant.create(subdomain: "test")
    load Rails.root + "db/seeds.rb"
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation, {except: config.seed_tables}
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end
end

Also, in case any of you find something wrong with this setup, please let me know. Thank you.

7

This database_cleaner config has evolved from Avdi’s. I’ve tried to bulletproof it. It’s worked fine with most setups I’ve encountered, when using javascript and non-javascript drivers:

8

Thanks for sharing your setup @eliotsykes. Can you please elaborate why you’d use type: :feature
instead of js: true . I see one reason mentioned in one of the comment’s in Avdi’s blog is -

Another side note, I use :type => :feature instead of :js => true since we don’t always need or remember to tag our specs as using javascript.

But doesn’t this have any performance downsides? Why would I want a truncation strategy for all feature specs especially those where no JS is involved? Kindly explain in case I’m missing something here.

Thank you.

9

Thanks for sharing your setup

You’re welcome!

What you describe isn’t exactly what’s in that file. I’ll try to summarize what is happening in that config:

  • If there’s a feature spec that’s using the :rack_test Capybara driver, then use database cleaner’s transaction strategy.
  • If there’s a feature spec that’s using any other Capybara driver (poltergeist and selenium drivers are both used in that sample app), then use the truncation strategy.

This means not all feature specs are using truncation, only those that use a non-rack_test driver (which will be all of the feature specs flagged with js: true).

Here’s the relevant part of the config:

  config.before(:each, type: :feature) do
    if Capybara.current_driver == :rack_test
      # :rack_test driver's Rack app under test shares database connection
      # with the specs, so we can use transaction strategy for speed.
      DatabaseCleaner.strategy = :transaction
    else
      # Non-:rack_test driver is probably a driver for a JavaScript browser
      # with a Rack app under test that does *not* share a database 
      # connection with the specs, so we must use truncation strategy.
      DatabaseCleaner.strategy = :truncation
    end
  end
10

Thank you for explaining this @eliotsykes . It’s nice to see the way you’ve explicitly specified that folks should consider specifying config.use_transactional_fixtures in the right way :+1:

IMHO, the below setup also behaves in the same way as your setup. If my Capybara feature spec doesn’t test for any JS it defaults to the transaction strategy as given in the first config.before(:each) block given below. Does using type: :feature provide any other additional benefits which I’m missing ?

  config.before(:suite) do
    DatabaseCleaner.clean_with :truncation
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

Thank you

11

Using js: true is fine for the majority of projects.

The reason I’ve avoided js:true and used :feature and test for the current driver is for 2 reasons:

Major reason For some of my specs, js: true isn’t enough as the specs use multiple different JavaScript-capable drivers (selenium-firefox and poltergeist) to test code that uses browser-specific APIs. For these specs I rely on the driver: ... spec metadata and not js: true.

Minor reason js: true is one level of abstraction higher than it needs to be which makes using it for this purpose a little inaccurate. The reason for using truncation is more closely related to the driver used, its not that the driver happens to support JavaScript.

12

@eliotsykes Thank you for explaining why you’re using type: :feature in your case. It’s more clearer now.