What are the popular libraries for writing integration level UI tests for iOS? I’ve got a rails background, so I am looking for something capybara for iOS. I would also prefer not to have to use cucumber.
Any ideas would be greatly appreciated.
The iOS guys at my place use KIF, with varying results, they’ve had problems setting slider touch events, for example
This comes up a lot. The tl;dr is that (in my opinion) all of the existing solutions kind of suck in one way or another. We (thoughtbot) haven’t been using integration/acceptance tests, instead doubling down on unit tests.
Now, to answer your question fully:
There are a few different solutions out there, each with its own set of issues:
Frank (Official Website)
Frank bills itself as Selenium for iOS apps. In order to use it, you inject a small framework into a special build of your application. This framework acts as a local server, which you can then interact with through the tests. Tests are written in Ruby and Cucumber, although I suppose it might be possible to use a framework other than Cucumber. I haven’t really tried.
Pros
- Frank uses a library that exposes the private UIAutomation framework so that you can access it directly. This has a benefit over some of the other libraries that synthesize their own touch events because it is leveraging Apple’s own infrastructure in order to drive the UI.
- Some people seem to like Cucumber and enjoy writing their tests in Ruby
Cons
- I, on the other hand, do not enjoy Cucumber. I also don’t like having to make a context switch when jumping from production code to testing code.
- Ruby seems like a fairly significant amount of overhead for new developers coming on the project. This could (read: probably will) lead to rot in the acceptance tests, which might even affect the unit tests (Broken Windows and all that)
- I’m unclear on the commitment to the continued development of the project. There are 2 different versions of each of the core libraries (Frank, PublicAutomation, Shelley) under different GitHub organizations (@moredip and @TestingWithFrank). It’s never clear which is the actual version. According to moredip/frank, the one under TestingWithFrank is the official version, but then PublicAutomation and Shelley will be updated in the private forks instead of in the “official” repos, and will stay out of sync for a time. It doesn’t help the confusion that the official website seems out of date and links to the repos under @moredip.
- Maintaining a different target just for acceptance testing is a not-insignificant amount of overhead, and is just fairly annoying. I believe there is a way to avoid having to use a different target, but then that introduces some other weird behavior with corrupted binaries due to the force injection of the framework.
All in all, Frank isn’t my cup of tea. I have friends in the community that have had success with Frank, but every time I’ve tried using it I’ve run into headache after headache.
KIF
KIF is a native Objective-C acceptance testing framework originally written by Square, no being led by the community. Of all of the acceptance testing frameworks available, this is probably the one I have the most experience with.
Pros
- Since you write tests with Objective-C, there isn’t any context switch needed jumping from production code to unit tests to acceptance tests. It’s all one language.
- The recent jump to KIF 2.0 made this even more seamless by working directly with XCTest, and integrating with popular Unit testing suites such as Specta and Kiwi.
- It encourages using the Actor model for testing, which is a nice level of abstraction for testing.
Cons
- As noted by @aaronmcadam, it can be inconsistent at times. This is a huge issue for any testing framework. If you can’t trust your tests, you won’t use them. False failures are constant with KIF
- I have real problems relying on their development cycle. As I stated before, this project was originally created by Square, which they then abandoned. After entirely too long, development was taken over by another individual, who then disappeared. It has since been taken over by yet another individual, and while he’s been good about getting PRs merged in, I’m skeptical about the direction the project is going in. Being a solo developer in charge of an open source library is exhausting and hard to maintain long term. Being a solo maintainer for a library of this size and complexity doesn’t seem like a great idea, and I am strongly under the opinion that it’s only a matter of time until this project is abandoned once again.
All in all, I think this is probably what you should look at if you really want to write tests in Obj-C, but I’ve been burned (and had clients get burned) enough that I can’t bring myself to trust it anymore.
Bwoken
Bwoken is a Ruby library for accessing the native UIAutomation APIs directly from Javascript/Coffeescript.
Pros
- Unlike everything else I’ve listed, I actually do trust it’s development. 2.1 RC 2 was just pushed 3 days ago. That kind of structure won’t be found with either of the other libraries.
- Is uses the public, official UIAutomation APIs. This is huge, since the other 2 rely on either synthesizing their own touch events or manually bridging to the private framework in Objective-C.
- You can run tests from the command line. In fact, I’m not sure that you have a choice here. I don’t see any way to be able to run tests directly from Xcode.
Cons
- Again, you write in Coffeescript/Javascript, so there’s a context switch in order to write the tests. This language requirement might also be an unreasonable amount of overhead for new devs.
- Documentation could probably be better. I don’t see much in terms of example usage on their GitHub or their website. That should be easily remedied, though.
Honestly, if I were starting a project today, I might choose Bwoken. I’m not thrilled at writing in Coffeescript, but given that the choice seems to be that or use one of these libraries that is hacking their way into the application and has a questionable development history, I’d bite the bullet and choose Bwoken.
3 Likes
Thanks @aaronmcadam and @gfontenot for your replies. That points me in a good direction.