Test Suite Speed

Testing
1

When practicing test-driven-development, the speed of your test suite can have a dramatic impact on your productivity. Some techniques for speeding up your tests are simple, such as using Factory Girl’s build_stubbed method instead of create. Others, such as disabling `bundler.require’ are more complex and require changes to your spec_helper and specs or your rails code itself.

We’ll be discussing ways to speed up your test suite and their associated pros and cons at the thoughtbot Boston developer discussion tomorrow afternoon and will post our notes and any follow-up here.

2

The best way I know for speeding up your test suite is to have your application core be independent of rails, that way you don’t need to load rails to test it, making it blazingly fast.

This, of course, is a very different way of doing things, which might be a big endeavour for most teams, but still, the more you can test without loading rails and/or hitting the db, the better.

You might of course have a different suite of integration tests, which by nature are slower.

1 Like
3

In my experiences, there are several different types of things that can slow down your tests. In no particular order and with some thoughts on how I try to combat them:

Feature Specs

  • Too many feature specs. I generally reserve feature specs for happy path tests of high value features (vs permutations of those features). Use view specs for view conditionals. Controller specs for simple sad-path tests.
  • Too much navigation. For instance, unless you aretesting logging in, don’t have your feature specs log in every time. Use the clearance backdoor or Devise’s somewhat clunkier equivalent.
  • JavaScript enabled feature specs: Capybara webkit is great, but these specs are super slow in comparison to Rack::Test. Use them when you need a full feature spec for a critical path in your app. Consider a JavaScript test suite if you need to test various paths through a script. Adding feature spec coverage for all your JavaScript is going to be super slow.

Too much database interaction:

  • Prefer MyObject.new over any factory use at all where feasible. Does your test only rely on the value of a single attribute? Just set it on initialization.
  • Prefer build_stubbed over build or create: Build stubbed will use your factories but stub out all of the persistance methods.
  • Get your factories under control. Do you really need create(:user) to add an associated blog every time or was that just convenient at one point? What if you did create(:blog).user to get that user record instead? What if your user factory had a with_blog trait?

** Configuration: **

  • Stay up-to-date with Ruby. Ruby 2.0 sped things up quite a bit. Ruby 2.1 a bit more.
  • Try disabling garbage collection in tests by adding GC.disable to your spec helper. I saw a good bit of speed up with this on some projects (less so on Ruby 2.0 than Ruby 1.9.3, haven’t tried it out on Ruby 2.1). I haven’t had any problems because of it, but I also make sure to exempt CI environments from this.
  • If you’re using the paper_trail gem for auditing, disable it in your spec_helper with PaperTrail.enabled = false. You can enable it specifically for individual specs if you want to ensure it’s working as expected. Disabling papertrail keeps it from creating hundreds of auditing records while your tests run. There may be other gems like this that can be configured for faster tests.
  • If you’re using bcrypt, be sure to set it’s DEFAULT_COST to the minimum value in your tests. Bcrypt is intentionally slow - we don’t want this behavior in tests. Most gems that use bcrypt (like Clearance and Devise) already take steps to handle this for you, but check it out if you’re using another gem or bcrypt yourself. See this blog post for a thorough explanation.

Those are things I can try on just about any project regardless of when I come onto it.

1 Like
4

We’ve used fast_spec_helper with success on an internal project.

5

@greg : tell us more. What was in fast_spec_helper ? Or perhaps more appropriately, what wasn’t?

6

Here’s the fast_spec_helper.rb:

$LOAD_PATH << File.expand_path('../..', __FILE__)

require 'webmock/rspec'

Dir['spec/support/**/*.rb'].each {|f| require f}

RSpec.configure do |config|
  config.order = 'random'
  config.include GithubApiHelper
  WebMock.disable_net_connect!(allow_localhost: true)
end

Then we explicitly require things we need in the spec file:

require 'rubocop'
require 'fast_spec_helper'
require 'app/models/style_checker'

describe StyleChecker, '#some_method' do
  context 'some condition' do

And in spec_helper.rb we require fast_spec_helper, to avoid duplication.

1 Like
7

That is pretty much what I meant by not loading rails.

There is also a variation of this, which is only loading ActiveRecord, popularised by Corey Haines: https://gist.github.com/coreyhaines/2068977

8

Here are the notes from today’s discussion:

Rails/Bundler Slowness:

  • bundle install --standalone option is similar to the speedup from Spring, according to @jferris’s experience. Joe was the only one of us to have tried both approaches.
  • Spring has improved since its early betas and will ship on by default in Rails 4.1.
  • Spring is enabled in Suspenders and better integration with FactoryGirl is in a pending PR.
  • We discussed the idea of using bundler --standalone as design feedback.
    • Explicitly listing requires will easily show you your dependencies and highlight smells.
    • Unfortunately, this isn’t enforced by Ruby. Unlike Java, if any file had previously required a library your are using, it’s now available to your file as well – even without a require. This is made even worse by Rails autoloading.
    • The idea is attractive in theory, but not really feasible in ruby.

** Separate Test Suites: **

Some people advocate for having separate test suites. One test suite that runs quickly and can give you 95% confidence and a second that you run to check everything out. We discussed this idea some.

  • For tests be successful, every single one of them has to pass before a merge is allowed.
  • CI should not be the only place that the full suite runs.
  • Split test suites result in “that other suite” becoming neglected. “Oh, those always fail [locally | on ci | for me]”
  • Studies show that if a test suite takes > 30s to run, developers won’t run them. If we have put so much effort into a fast test suite that we can say “We need all this coverage and there’s no way we can get it under 30 seconds” then maybe we should consider splitting it up, but it’s unlikley on our smaller apps we’ve ever putt enough effort into that to reach that.
  • Exceptions to this are for gems that test on multiple versions of ruby and with multiple dependencies. This is too much of a hassle to do locally. Appraisal can help, but it still requires manually switching rubies, etc.

** The Testing Pyramid **

We discussed the testing pyramid and the rails testing pyramid. The big takeaway is that we should be continually trying to push tests down to the faster, wider levels of the pyramid. Also, we should be careful to not automatically categorize model tests as fast or controller specs as slow, as reality is often opposite (see tests of ActiveRecord scopes).

We also discussed the contention (from the code climate article) that you should have only 7 feature specs. There wasn’t anyone that spoke up in support of this idea. You should have as many feature specs as makes sense for your app, but you should be questioning whether the feature spec you wrote to drive out a feature has ongoing value or if it could be replaced by lower level tests. It’s okay to throw away a feature spec and replace it with coverage at other layers.

** Speed Up Strategies **

We also discussed some quick tips to help speed up tests. We agreed on these:

  • Use .new instead of factories. If you need the factory, use build_stubbed instead of build or create.
  • If you use FactoryGirl’s *_list methods (e.g. create_list), keep the number of items at two. create_pair was recently added (perhaps not yet released) to FactoryGirl to encourage this practice over large lists.
  • stub all external requests using with webmock, vcr, etc, or use a Fake in tests.
  • Write more view specs: don’t test conditionals in your feature specs.
  • Write more controller specs: test error handling there. Consider render views to ensure that errors are handled and rendered properly (versus just checking that the flash was set). This will be more involved than a regular controller spec, but will be less involved than a feature spec.
  • Use your authentication frameworks backdoor when you aren’t actually testing sign in/up.
  • Watch out for gems that do costly things automatically: Disable paper_trail in tests. Stub paperclip in your tests.
  • If you’re using BCrypt in your app, double check that the default cost is cranked down to the minimum.

There were some concerns raised about GC.disable. This will lead to faster tests early and with smaller apps and suites, but as the suite grows this may cause problems. Of course, you could run out of memory (I hope not… get more RAM), but more likely your Ruby process will now have enough active RAM that execution will actually slow down. If you do add GC.disable be sure to disable it on your CI, and periodically check its effectiveness in your dev setup.

2 Likes
9

@derekprior Great tips on speeding up test suites.

Especially the ones about checking errors in controllers and stubbing paperclip. Don’t hear that too often.

Thanks.