Here are the notes from today’s discussion:
- bundle install --standalone option is similar to the speedup from Spring, according to @jferris’s experience. Joe was the only one of us to have tried both approaches.
- Spring has improved since its early betas and will ship on by default in Rails 4.1.
- Spring is enabled in Suspenders and better integration with FactoryGirl is in a pending PR.
- We discussed the idea of using bundler --standalone as design feedback.
- Explicitly listing requires will easily show you your dependencies and highlight smells.
- Unfortunately, this isn’t enforced by Ruby. Unlike Java, if any file had previously required a library your are using, it’s now available to your file as well – even without a require. This is made even worse by Rails autoloading.
- The idea is attractive in theory, but not really feasible in ruby.
** Separate Test Suites: **
Some people advocate for having separate test suites. One test suite that runs quickly and can give you 95% confidence and a second that you run to check everything out. We discussed this idea some.
- For tests be successful, every single one of them has to pass before a merge is allowed.
- CI should not be the only place that the full suite runs.
- Split test suites result in “that other suite” becoming neglected. “Oh, those always fail [locally | on ci | for me]”
- Studies show that if a test suite takes > 30s to run, developers won’t run them. If we have put so much effort into a fast test suite that we can say “We need all this coverage and there’s no way we can get it under 30 seconds” then maybe we should consider splitting it up, but it’s unlikley on our smaller apps we’ve ever putt enough effort into that to reach that.
- Exceptions to this are for gems that test on multiple versions of ruby and with multiple dependencies. This is too much of a hassle to do locally. Appraisal can help, but it still requires manually switching rubies, etc.
** The Testing Pyramid **
We discussed the testing pyramid and the rails testing pyramid. The big takeaway is that we should be continually trying to push tests down to the faster, wider levels of the pyramid. Also, we should be careful to not automatically categorize model tests as fast or controller specs as slow, as reality is often opposite (see tests of ActiveRecord scopes).
We also discussed the contention (from the code climate article) that you should have only 7 feature specs. There wasn’t anyone that spoke up in support of this idea. You should have as many feature specs as makes sense for your app, but you should be questioning whether the feature spec you wrote to drive out a feature has ongoing value or if it could be replaced by lower level tests. It’s okay to throw away a feature spec and replace it with coverage at other layers.
** Speed Up Strategies **
We also discussed some quick tips to help speed up tests. We agreed on these:
.new instead of factories. If you need the factory, use
build_stubbed instead of
- If you use FactoryGirl’s
*_list methods (e.g.
create_list), keep the number of items at two.
create_pair was recently added (perhaps not yet released) to FactoryGirl to encourage this practice over large lists.
- stub all external requests using with webmock, vcr, etc, or use a Fake in tests.
- Write more view specs: don’t test conditionals in your feature specs.
- Write more controller specs: test error handling there. Consider
render views to ensure that errors are handled and rendered properly (versus just checking that the flash was set). This will be more involved than a regular controller spec, but will be less involved than a feature spec.
- Use your authentication frameworks backdoor when you aren’t actually testing sign in/up.
- Watch out for gems that do costly things automatically: Disable
paper_trail in tests. Stub paperclip in your tests.
- If you’re using BCrypt in your app, double check that the default cost is cranked down to the minimum.
There were some concerns raised about
GC.disable. This will lead to faster tests early and with smaller apps and suites, but as the suite grows this may cause problems. Of course, you could run out of memory (I hope not… get more RAM), but more likely your Ruby process will now have enough active RAM that execution will actually slow down. If you do add
GC.disable be sure to disable it on your CI, and periodically check its effectiveness in your dev setup.