Create your own app – Part 4: App and Usability Testing

As soon as an app is completed, the next step is app testing. The app is tested for functionality and usability as well as its compatibility with different devices and operating systems.

Most developers want of course want to publish their application as soon as possible after developing it and setting the final app design – but before the software is offered in the app store, it is important to test its suitability across various categories. Common methods for determining this are performance, compatibility, and usability tests. These and other testing methods are grouped together under the term “mobile application testing” (or “mobile app testing” for short). Different app testing tools are used throughout the various stages of this process.

In order for an app to be successful, it not only needs to offer impressive content and be well-marketed, but above all must work flawlessly and be user-friendly. Therefore, fundamental testing before the release of an application is vital – whether it is a mobile app or a desktop application.

Testing apps is, however, becoming more difficult because of one increasingly complex factor: adaptation to different user devices is considerably more elaborate for apps than for other software. In the early years of app development this adaptation was easier to manage due to the relatively small selection of smartphones on the market. Since then, a huge selection of mobile devices has emerged with a variety of different features. Most smartphone and tablet models function on roughly the same quality of hardware, but installed components such as the processor (CPU), memory (RAM), screen size, and screen resolution are the main factors determining the rendition of an application.

In addition to its hardware, the software of a device also plays a role. Software mainly contains the different operating systems of a platform, which influence app rendition. Older operating systems can make app usage more difficult due to missing updates, and the prevalence of old Android and iOS operating systems should not be underestimated. The variety of operating systems in use is particularly evident when looking at Android systems: On the whole, the older versions of Google’s operating system are used more commonly than the current version. This has to do with the fact that many Android manufacturers either deliver updates late, or don’t deliver them at all. With iOS the new version of the operating system is generally delivered more quickly, but some of the oldest devices no longer receive updates from Apple, therefore old iOS versions can still be found.

The variety of hardware and software for mobile devices is called fragmentation and it presents many challenges when it comes to app quality assurance. This is where mobile app testing comes into play: This checks whether an application is even capable of running under different conditions, and if so, whether the conditions present any difficulties for playback of the app. Detecting and correcting such issues is necessary to ensure an optimal user experience for as many users as possible. Only when an app offers good performance and ease of use for as many devices as possible can it reach its maximum potential.

App testing is not all done the same way. There are many different approaches to quality assurance. The two main questions to address are who should check the quality of the application (either manual or automated tests) and how the tests should be conducted (tests directly on the respective devices or tests via a simulator or emulator).

Manual app tests (carried out by users)

The manual test is the classic approach to mobile app testing – in this case, real people test the application in question. Since the testing of applications can be quite extensive (especially with Android apps), the development team usually cannot do this by themselves. Consequently, test subjects are used. A group is assembled (representative of a larger group, in the best case scenario) which then uses the app according to predetermined test scenarios and subsequently evaluates it. Outside test subjects have the advantage that they do not have any prior knowledge of how the application should work, and therefore can evaluate with judgment close to that of a potential user.

In addition to the manual mobile app testing, crowd testing is another classic approach. Crowd testing services offer a pool of testers, which allows developers to choose targeted test persons to test the app on their own personal devices. Crowd testing does not take place in a special testing environment, but rather in the natural environment of the users, which can increase the validity of the test results. For iOS apps (as well as watchOS and tvOS apps) developers can use the TestFlight Beta Testing service to invite up to 2,000 users to test the application and organize their own crowd tests. Google also offers software developers various options for private and public beta testing of Android apps.

Automated app tests (carried out by app testing tools)

In contrast to manual tests, automated controls are used in areas where certain processes are continuously tested under different conditions. For automated app tests, developers need to use a special software. The main advantage is how much time it saves – test procedures can carry out programs in a fraction of the time that a user would need. However, automation is not suitable for all test fields.

Tests on physical mobile devices

Testing apps on individual mobile devices is the ideal solution for being able to understand the app in combination with specific software and hardware. However, this is quite expensive and time-consuming, because a large number of smartphones and tablets are required for comprehensive tests, and the behavior of the app has to be constantly be checked in detail on each device. The easiest way to test using this method is by using various mobile app testing vendors, which provide access to a variety of mobile devices (often called “device lab” or “test lab”) that can be used to test the app.

Tests via Simulator/Emulator

Simulators and Emulators are programs that imitate particular devices and allow applications to be tested virtually. However, the results obtained from these tests are not always 100% reliable, as not all user interactions can be authentically simulated. In addition, some tests should be carried out on physical devices as the simulation can reach its limits in certain areas.

Simulators and emulators can admittedly not replace tests on physical devices, though they are a good addition and particularly suitable as an automated testing tool. With their help, it is possible to quickly test the functionality of an app with different hardware and software combinations. Many simulators and emulators are also available for free – for example, those contained within the Android Studio and Apple’s Xcode programs.

In addition to the various methods for testing an app, there are also various areas of testing. Apart from the tests mentioned in this section, there is also safety testing, security testing, and localization testing, as well as other test types and areas.

By using functionality tests, a developer can check that the app is really working the way they want it to. Ultimately, all elements of the app will be checked here for correct operation – every function, screen, and button will be inspected. Functionality testing focuses on the following central questions:

• Is the app easily installed and uninstalled?

• Do all app elements function as intended?

• Can the app continue to run smoothly when minimized?

• Does the app crash?

• Does the user receive an appropriate error message when an error occurs?

In addition to the app itself, tests should check the behavior of the application in the certain following scenarios:

• Can other applications and functions of the device be carried out when the app is minimized?

• Does the app automatically minimize when a call is received?

• Are SMS and other notifications received and stored during app usage?

App usability testing checks how user-friendly the operation of the application is. Automated tests are not an option here – after all, only users can assess whether the how well they think an app works. The use of a simulator or emulator is also generally not helpful for usability testing, as app operation can only truly be evaluated by usage on a mobile device; only when using the actual application can control procedures such as swiping, scrolling, and pressing buttons be felt and their reaction times and behaviors be evaluated. Other aspects that are examined in usability testing are as follows: 

• Is operation of the app largely self-explanatory? Are explanations provided for more complicated functions?

• Are app elements well placed, and important functions easily located by the user?

• Is the placement of elements and layout of the user interface organized in a way that is intuitive and easy to understand?

• Are the buttons optimally positioned and large enough to accommodate users with wider fingers?

• Are texts clearly written and formatted in a way that’s easily readable and appropriately sized?

Usability testing should discover any problems with the operation of the app or shortcomings that detract from its ease of use. In order for the results to be conclusive, the usability test must be carried out by a sufficient number of test users. It is also advantageous if the test users (at least a majority) are people from the target group – that way the operation and structure of the app can be optimized for expected users. The resulting improvements allow the developer to offer the best possible user experience.

The compatibility of an app in regard to the hardware and software of mobile devices should always be tested. In terms of hardware, one important example is testing the rendering of the application on different displays. Because there are different screen sizes and resolutions, display and customization errors can occur relatively quickly, especially if several layouts of the application for different screen formats have not already been created. In addition to the display format, CPU and RAM vary from device to device. Keep in mind that older mobile devices are not a rarity. Apps that are designed not only for current models, but also for older smartphones and tablets, automatically increase their market range.

In regard to device software, the smooth playback of an application is largely controlled by the different versions of the operating system – Android app testing on various versions of the operating system is especially important, as this platform has significantly more versions with more variation of use than Apple’s iOS platform.

Compatibility testing allows developers to check which hardware or software processes cause immediate problems with rendition. If the requirements for the device components are very high, it may be helpful to reduce certain system requirements in order to avoid shrinking the target group.

Performance tests show how many system resources an application consumes during intense use. Does the app continue to run smoothly, or does it experience lapses? Beyond that, performance tests can determine if a device’s battery will become overloaded or if the device itself will overheat after a certain period of use. As with compatibility testing, tests on the performance of the applications are controlled under certain conditions – for example, how the playback of the app changes if there is hardly any space left on the device’s hard drive or if the battery is almost empty. Performance testing is intended to reveal the extent to which certain circumstances negatively impact the app’s performance.

If one or more components are going to be subject to high utilization, then a load test (or stress test) is used. One example of this is the server stress test, where the app’s behavior is tested in reaction high server demand. For applications that require an internet connection, a network load test is conducted to see how a weak internet connection affects app functionality. Stress tests are designed to show how much stress certain components can take before their performance becomes restricted.

The above tests don’t necessarily have to be carried out manually. Diverse testing tools can make the work considerably easier in many areas of app testing. Seeing as this article series discusses mobile application development for both Android and iOS, the following app testing tools are those which are consistently available across both platforms.

Appium is a cross-platform open source program for test automation that is available for download free of charge. This tool doesn't only test native Android and iOS apps, but also supports test automation of hybrid apps and web apps.

Appium uses the same programming interface for both iOS and Android, so it is easy to reuse previously written code. The created tests can be carried out on physical devices, simulators, or emulators. Appium is incredibly helpful software for the execution of automated app tests, however, the program does require the user to have a certain level of pre-existing know-how.

On TestFairy’s website, developers can upload their app and invite people to participate in app testing. These people can then download and use the application – essentially, TestFairy allows for self-organized crowd testing. The tool has an error reporting feature and can record video of app test usage. TestFairy is available as a free download and also a paid enterprise version.

Ubertesters is a platform that helps with beta testing for apps. Developers have complete control over the tests and receive detailed reports. On the platform it is easy to work together as a team, detect programming errors using various tools, create and test different versions of the app, and perform crowd tests (for example, in the form of functionality or usability tests). Ubertesters is offered in various packages, each with different features – the package with the fewest features is available for free download.

Obviously, the reach of an app can be extended by designing it to run on many older devices. The adaptation of an app to the largest possible number of devices, though, has its limits. CPU-intensive applications, for example, whose functions require a stronger processor and/or more memory, can sometimes not be prepared optimally for older hardware. Another example: if an app requires a powerful internet connection, it should absolutely not be adapted to any usage situation in which there is only a weak internet connection.

Some applications simply require certain technical prerequisites. For example, some applications require a particular screen size in order to run. Another example is the YouTube app, which can only function with access to a sufficiently fast data transfer rate; yet it is still one of the most downloaded applications ever. Adjustments and reductions of functionality in favor of a large app range should only be made if the app functionality is not reduced more than desired.