Investigating inconsistently failing tests

Testing
1

Hello everyone!

Let’s say you faced this situation on the large codebase :smile:

What is the strategy for investigating inconsistently failing tests?

Quick thoughts:

  1. Be able to re-run the tests in the same order as when it failed (default is random)
  2. Database snapshot at the point of failure (already have DatabaseCleaner in place)?

Thank you in advance!

2

I find that inconsistently failing tests are usually caused by one of two things:

  • State leaking between tests
  • Asynchronous timing problems

The first is usually easier to solve.

If you’re using RSpec, you can provide a seed value to the rspec command which will cause the tests to be run in the same order each time:

Also, RSpec 3.3 added a bisect command which will repeatedly run your tests until it can determine which combinations are causing a failure:

Being able to inspect the state of the app the point of the failure is certainly useful - I’d suggest using the pry-rescue gem for this:

2 Likes
3

Nice talk on the subject: