This post highlights some of the key issues that inevitably surface when dealing with iOS mobile application testing.

Don't Underestimate the Complexity of iOS

As a closed mobile operating system, iOS poses challenges for mobile testing. Since test developers do not have access to the application's native objects, test automation becomes much more difficult than testing on an Android or other open-source based OS platform.

While there's no doubting the popularity of iOS, it should be noted that not all iOS versions were created equal. In fact, this operating system has become increasingly fragmented - currently supporting six different screen resolutions for tablets, smartphones and iPods. Applications must also take into account multiple minor/major quarterly iOS releases. Furthermore, iOS supports various mobile web browsers, which need to be considered when testing web-based applications (Safari (default), Dolphin, Opera, Chrome, etc.).The screenshot below illustrates an example of iOS complexity specific to the iPhone4S device.

Figure 1: iOS 6.1 bug on iPhone 4S (Source: LINK)

Such a defect could have been detected by application developers prior to the commercial version release if they had started testing some of the iOS6.1 preliminary Beta versions.

These preliminary versions are usually made available to developers in advance (e.g., in few weeks the new iOS6.1.1 version is going to be released - a beta download which can be used for regression testing is already available).

A Word about Objects

There has been an ongoing debate in the iOS mobile testing ecosystem about the best approach to object testing – i.e., the use of jailbroken iOS devices vs. non jailbroken devices.The recent ruling that jailbreaking is a violation of the DMCA with respect to iPads – but not for iPhones - has further complicated the execution of mobile testing for iOS. Gaining access to the application objects is a prerequisite for test automation – and the development of efficient test automation is predicated upon access to OS level object IDs (application buttons, etc.).For testing mobile apps, visual analysis capabilities are particularly important due to the differences in screen size and resolution across mobile devices. For instance, you may be able to identify an object in your application on an iPhone4S, but the same object truncates or does not look good on an iPad or iPod. The only way you can identify such an issue is by using visual analysis (OCR).

To put it simply, in order to efficiently develop robust cross iOS device automation, your test automation solution needs to support:

- Non jailbroken iOS devices

- OS level (native) objects

- OCR (visual analysis)

The main obstacle in working with OS level objects in iOS is the need for an instrumented iOS application (compiled application with additional library) to fetch these objects from the native OS level.In light of the above, the natural conclusion is that your mobile application testing solution (BTW not just iOS) should support both visual analysis and OS level objects within the same script on non jailbroken devices.

The following table summarizes the pros and cons of OCR and OS level object analysis:

Figure2: Visual analysis (OCR) vs. Object level analysis – Pro’s and Con’s

To meet the ever-increasing demands of today's iOS users, application developers require the ability to test their iOS apps on any non jailbroken device, using any iOS version (including the pre-launched Beta 6.1.1 mentioned above) and without the need for instrumentation.

Figure3: Example of a native iOS testing solution on a non jailbroken iPhone5 

Author: Eran Kinsbruner is a Director at Perfecto Mobile, the leading cloud-based mobile testing, automation and monitoring company based in Woburn, MA. You can reach him at erank@perfectomobile.com.