iOS Feature Testing

A comparison of the current iOS feature testing frameworks available.

Features

Comparison overview of each of the frameworks' features such as UIControl interaction, remote processes, and the ability to be run from the command line.

Essential

UIKit Interactions

Hybrid Apps

External to App

Developer Niceties

💚 = Full support 💛 = Support with caveats

Rubric

A more detailed look into the pros and cons of each framework.

1. How easy it is to write tests first
2. Familiarity with language tests are written in
3. Ability to be run via the command line and on CI
4. Reliability
5. Coverage
6. How close to real interaction it is
7. Documentation
8. Community
9. Framework maintenance
10. Ability to run on physical devices (nice to have)

Frank

1. Developer can target UI elements via css-like selectors. This frees the developer from having to know ahead of time the implementation of the view hierarchy.
2. Tests are written in Ruby with lots of Cucumber helper steps optionally available.
3. There is a single command to "Frankify" the app, but CI setup takes more custom work.
4. Some tests can become flaky if written poorly. Asynchronous behavior quickly becomes unreliable, especially with animations and chaining events.
5. Most user interactions are able to be reproduced. The exceptions are MFMailComposeViewController, ABPeoplePickerNavigationController, and other controllers that aren't actually handled inside the app (eg. asking for location permissions and access to the address book). There are workarounds for some but not all of these issues.
6. The framework uses UIAutomation when possible but some interactions are not as "true" as they could be. Lot's of Frank's magic comes from UIKit objects being extended. For example, swiping views is all done in Objective-C to workaround it not being available in UIAutomation.
7. There is a decent amount of setup and implementation details on the website but lots of the functionality isn't documented or explained.
8. frank-discuss Google group is more of a knowledge base than an active discussion forum. Stack Overflow is lacking information on Frank.
9. Frank is maintained so poorly we have considered forking it and maintaining it on our own. This, however, has improved since iOS 8 has launched.
10. Frank claims to be able to run on devices but I haven't gotten it to work.

UIAutomation

1. UIAutomation requires that each level of the view hierarchy be explicitly referenced. For example, to access a button in a scroll view your syntax would look something like UIATarget.localTarget().frontMostApp().mainWindow().scrollViews[0].buttons()[0]. This makes your feature tests tied to the implementation of your view hierarchy.
2. UIAutomation is written in JavaScript. There are a number of wrappers built on top of it to help make it less verbose, such as bwoken and tuneup_js.
3. It is possible to run UIAutomation scripts via the command line with the some custom build scripts. The main idea is to inject the JavaScript into the app and then run it via Instruments.
4. While reliable when running via the GUI and command line, there is a race condition bug in Instruments that occurs on lower-end machines. This makes it particularly difficult to use when running as CI on a Mac Mini.
5. UIAutomation should be able to cover 99% of all use cases when running on a mobile device. However, a portion of these are broken with the introduction of iOS 7 (and continue to remain broken in iOS 8). These range from basic features such as scrolling views to very complex interactions like putting the app to sleep for five seconds.
6. Being UIAutomation is from Apple it should be as close to “real” user interaction as possible.
7. Most of Apple’s documentation exists in the UI Automation JavaScript Reference pdf.
8. There is a fair amount of talk online regarding UIAutomation and even a book from Pragmatic Bookshelf, Test iOS Apps with UI Automation: Bug Hunting Made Easy, covering how best to use.
9. Apple owns it and it’s closed source. It doesn’t appear that any updates have occurred since iOS 6.
10. Works but takes an extra couple of seconds to get installed on the device.

Subliminal

1. Similar to KIF, all views are interacted with via their accessibility labels. Subliminal traverses the view hierarchy looking for each match, no knowledge the way the views are laid out is needed.
2. Tests are written in Objective-C as subclasses of XCTest.
3. While no issues are apparent in Subliminal itself, the tests are run through Instruments which carries its own slew of problems. See UIAutomation’s notes on Reliability for more information.
4. Subliminal has the best coverage off all the frameworks, even better than UIAutomation itself. Most of the broken parts of UIAutomation have been patched (either in Objective-C or JavaScript directly) and work well. One of the few remaining limitations is lack of interaction with WKWebView.
5. Beings it runs UIAutomation scrips under the hood, interaction should be as close to “real” user interaction as possible.
6. Every public method is documented with expectations and parameters with full integration into Dash.
7. Despite its shortcomings Subliminal seems to have the largest community of users. Twitter even publicly acknowledged using it.
8. There haven’t been many recent commits to master and the Xcode 6 / iOS 8 branch has been open for almost six months.
9. Subliminal claims to be able to run on devices but I haven't gotten it to work.

KIF

1. All views are interacted with via their accessibility labels. Since KIF traverses the view hierarchy looking for each match, no knowledge the way the views are laid out is needed.
2. Tests are written in Objective-C as subclasses of XCTest.
3. Tests are run with the standard xcodebuild test command.
4. KIF is the most reliable of all the frameworks tested.
5. Similar to Frank, most user interactions are able to be reproduced. KIF still suffers from the same issues that plague Frank (e.g. it can’t interact with views outside of the app’s control). However, with recent additions it now can interact with system alerts (e.g. location services and photo authorization dialogs).
6. Most (if not all) of KIF’s interaction boils down to create UITouch events at the right point and sending them to the app.
7. Every public method is documented with expectations and parameters with full integration into Dash.
8. There is a Google Group which only gets about one post per week. The same goes for the official StackOverflow tag, kif-framework.
9. Framework is actively maintained with PRs being addressed within a few days of submission.
10. Works for all cases except interacting with system alerts.

Footnotes

• ¹ Some animations are handled without interaction, while others require manual waiting.
• ² Sometimes Frank is so eager to type, he doesn’t wait for the UITextField to fully focus, leading to dropped characters. Workarounds are possible.
• ³ Subliminal can slide a slider and successfully call delegate methods, but the computation of the physical nub offsets are left to you.
• ⁴ UIAutomation makes each UIPicker selection on value at a time, making selection very slow. Also, if the date is not selectable UIAutomation will silently fail.\
• ⁵ Subliminal cannot interact with these elements directly, but can call into UIAutomation’s JavaScripts via SLTerminal -eval:.
• ⁶ Kif itself isn't a BDD framework, but integrates seamlessly with BDD frameworks such as Quick.
• ⁷ Subliminal loops over Objective-C code which calls JavaScript asynchronously via Instruments making debugging possible, but quite difficult.
• ⁸ Tests can be focused in Xcode via tapping the gutter (ala XCTest) but not via the command line.
• ⁹ You can use the Accessibility Inspector to identify elements for KIF to interact with, but there isn’t a direct way to view the hierarchy or identify elements which KIF ignores because they are accessibility containers.

Versions

• Xcode 6.1
• iOS 8.1
• Frank 1.2.3
• Subliminal 1.1.0 - shared/Xcode6 branch - d99fef4 commit
• KIF - 3.0.8