Weird situation: tests pass and fail randomly

Testing
1

Hello.

I’ve been working on a OS project for the past weeks, and I ran into a wird situation where tests pass or fail randomly. It used to be just one of them, always the same, but now, after some more work, I’ve even had the situation where sometimes two tests fail, run again and one fails, run again and all pass, etc.

Have you ever been in such a situation? I’m not really looking for help on solving this, but rather know if this is something that other people come across, because I must say that it is actually the second project in which this happens to me.

Just for reference, no database involved or anything.

Thank you.

2

yes I have run into it. Most of the time it usually is when the tests are dealing with times or dates and were not written properly.

3

Hello Jared. Thanks for your reply. Now I know that this might be more common that I thought. I’ll double check, but I think that no dates are being used. Thanks!

4

By default, minitest will run your tests in a random order each time. It may be that some of the tests are dependent on a previous test having already run.

To check if this is the case, you can temporarily enable the (ridiculously named) i_suck_and_my_tests_are_order_dependent! setting.

5

LOL. I’ll try that. I really hope my tests are still failing after that :joy:

Thank you for pointing this.

6

So… I’m actually starting to think that it has something to do with the order the test suite is run, so I guess I suck :cry:

I’ve noticed that data in one test affects other tests, which is really weird…

For instance, I have this in one test file:

def test_that_unknown_extensions_produce_an_error                        
  serializer = UserSerializer.new(@user)                                 
  serializer.class._extensions = [:non_existing_extension] # <--- this causes trouble later
         
  assert serializer.errors.include?(:extensions)                         
end

Now, in one of the tests that fail randomly, I put an inspect on extensions just to see what is going on:

def test_response_format_with_one_resource                               
  user_serializer = UserSerializer.new(@user1)                           
  
  puts user_serializer.extensions.inspect
  # ...
end

user_serializer.extensions is an empty array, as expected, because the fixture UserSerializer does not use them. However, sometimes it has [:non_existing_extension], which was used in another test, as I showed.

I’ve tried to just set serializer.class._extensions to an empty array after the former test is run, and yes, then everything works fine.

I don’t yet understand how this is even possible, but I feel I’m about to learn something important today. Hope I won’t suck that much after that :stuck_out_tongue_winking_eye:

7

Yep, when you make changes to UserSerializer at runtime they will stick around since classes aren’t reloaded between test cases.

8

Cool. I didn’t know that. Good lesson to learn!

Thank you for your help.

9

Best method name ever! lol

1 Like
10

Actually I’m pretty sure this one is.

11

LOL Nice! Good to know that’s in sidekiq lol