Create your own app – Part 4: App and Usability Testing

As soon as an app is completed, the next step is app testing. The app is tested for func­tion­al­i­ty and usability as well as its com­pat­i­bil­i­ty with different devices and operating systems. Most de­vel­op­ers want of course want to publish their ap­pli­ca­tion as soon as possible after de­vel­op­ing it and setting the final app design – but before the software is offered in the app store, it is important to test its suit­abil­i­ty across various cat­e­gories. Common methods for de­ter­min­ing this are per­for­mance, com­pat­i­bil­i­ty, and usability tests. These and other testing methods are grouped together under the term “mobile ap­pli­ca­tion testing” (or “mobile app testing” for short). Different app testing tools are used through­out the various stages of this process.

In order for an app to be suc­cess­ful, it not only needs to offer im­pres­sive content and be well-marketed, but above all must work flaw­less­ly and be user-friendly. Therefore, fun­da­men­tal testing before the release of an ap­pli­ca­tion is vital – whether it is a mobile app or a desktop ap­pli­ca­tion.

Testing apps is, however, becoming more difficult because of one in­creas­ing­ly complex factor: adap­ta­tion to different user devices is con­sid­er­ably more elaborate for apps than for other software. In the early years of app de­vel­op­ment this adap­ta­tion was easier to manage due to the rel­a­tive­ly small selection of smart­phones on the market. Since then, a huge selection of mobile devices has emerged with a variety of different features. Most smart­phone and tablet models function on roughly the same quality of hardware, but installed com­po­nents such as the processor (CPU), memory (RAM), screen size, and screen res­o­lu­tion are the main factors de­ter­min­ing the rendition of an ap­pli­ca­tion.

In addition to its hardware, the software of a device also plays a role. Software mainly contains the different operating systems of a platform, which influence app rendition. Older operating systems can make app usage more difficult due to missing updates, and the preva­lence of old Android and iOS operating systems should not be un­der­es­ti­mat­ed. The variety of operating systems in use is par­tic­u­lar­ly evident when looking at Android systems: On the whole, the older versions of Google’s operating system are used more commonly than the current version. This has to do with the fact that many Android man­u­fac­tur­ers either deliver updates late, or don’t deliver them at all. With iOS the new version of the operating system is generally delivered more quickly, but some of the oldest devices no longer receive updates from Apple, therefore old iOS versions can still be found.

The variety of hardware and software for mobile devices is called frag­men­ta­tion and it presents many chal­lenges when it comes to app quality assurance. This is where mobile app testing comes into play: This checks whether an ap­pli­ca­tion is even capable of running under different con­di­tions, and if so, whether the con­di­tions present any dif­fi­cul­ties for playback of the app. Detecting and cor­rect­ing such issues is necessary to ensure an optimal user ex­pe­ri­ence for as many users as possible. Only when an app offers good per­for­mance and ease of use for as many devices as possible can it reach its maximum potential.

App testing is not all done the same way. There are many different ap­proach­es to quality assurance. The two main questions to address are who should check the quality of the ap­pli­ca­tion (either manual or automated tests) and how the tests should be conducted (tests directly on the re­spec­tive devices or tests via a simulator or emulator).

Manual app tests (carried out by users) The manual test is the classic approach to mobile app testing – in this case, real people test the ap­pli­ca­tion in question. Since the testing of ap­pli­ca­tions can be quite extensive (es­pe­cial­ly with Android apps), the de­vel­op­ment team usually cannot do this by them­selves. Con­se­quent­ly, test subjects are used. A group is assembled (rep­re­sen­ta­tive of a larger group, in the best case scenario) which then uses the app according to pre­de­ter­mined test scenarios and sub­se­quent­ly evaluates it. Outside test subjects have the advantage that they do not have any prior knowledge of how the ap­pli­ca­tion should work, and therefore can evaluate with judgment close to that of a potential user. In addition to the manual mobile app testing, crowd testing is another classic approach. Crowd testing services offer a pool of testers, which allows de­vel­op­ers to choose targeted test persons to test the app on their own personal devices. Crowd testing does not take place in a special testing en­vi­ron­ment, but rather in the natural en­vi­ron­ment of the users, which can increase the validity of the test results. For iOS apps (as well as watchOS and tvOS apps) de­vel­op­ers can use the Test­Flight Beta Testing service to invite up to 2,000 users to test the ap­pli­ca­tion and organize their own crowd tests. Google also offers software de­vel­op­ers various options for private and public beta testing of Android apps. Automated app tests (carried out by app testing tools) In contrast to manual tests, automated controls are used in areas where certain processes are con­tin­u­ous­ly tested under different con­di­tions. For automated app tests, de­vel­op­ers need to use a special software. The main advantage is how much time it saves – test pro­ce­dures can carry out programs in a fraction of the time that a user would need. However, au­toma­tion is not suitable for all test fields.

Tests on physical mobile devices

Testing apps on in­di­vid­ual mobile devices is the ideal solution for being able to un­der­stand the app in com­bi­na­tion with specific software and hardware. However, this is quite expensive and time-consuming, because a large number of smart­phones and tablets are required for com­pre­hen­sive tests, and the behavior of the app has to be con­stant­ly be checked in detail on each device. The easiest way to test using this method is by using various mobile app testing vendors, which provide access to a variety of mobile devices (often called “device lab” or “test lab”) that can be used to test the app.

Tests via Simulator/Emulator

Sim­u­la­tors and Emulators are programs that imitate par­tic­u­lar devices and allow ap­pli­ca­tions to be tested virtually. However, the results obtained from these tests are not always 100% reliable, as not all user in­ter­ac­tions can be au­then­ti­cal­ly simulated. In addition, some tests should be carried out on physical devices as the sim­u­la­tion can reach its limits in certain areas.

Sim­u­la­tors and emulators can ad­mit­ted­ly not replace tests on physical devices, though they are a good addition and par­tic­u­lar­ly suitable as an automated testing tool. With their help, it is possible to quickly test the func­tion­al­i­ty of an app with different hardware and software com­bi­na­tions. Many sim­u­la­tors and emulators are also available for free – for example, those contained within the Android Studio and Apple’s Xcode programs.

In addition to the various methods for testing an app, there are also various areas of testing. Apart from the tests mentioned in this section, there is also safety testing, security testing, and lo­cal­iza­tion testing, as well as other test types and areas.

By using func­tion­al­i­ty tests, a developer can check that the app is really working the way they want it to. Ul­ti­mate­ly, all elements of the app will be checked here for correct operation – every function, screen, and button will be inspected. Func­tion­al­i­ty testing focuses on the following central questions:

• Is the app easily installed and unin­stalled?

• Do all app elements function as intended?

• Can the app continue to run smoothly when minimized?

• Does the app crash?

• Does the user receive an ap­pro­pri­ate error message when an error occurs?

In addition to the app itself, tests should check the behavior of the ap­pli­ca­tion in the certain following scenarios:

• Can other ap­pli­ca­tions and functions of the device be carried out when the app is minimized?

• Does the app au­to­mat­i­cal­ly minimize when a call is received?

• Are SMS and other no­ti­fi­ca­tions received and stored during app usage?

App usability testing checks how user-friendly the operation of the ap­pli­ca­tion is. Automated tests are not an option here – after all, only users can assess whether the how well they think an app works. The use of a simulator or emulator is also generally not helpful for usability testing, as app operation can only truly be evaluated by usage on a mobile device; only when using the actual ap­pli­ca­tion can control pro­ce­dures such as swiping, scrolling, and pressing buttons be felt and their reaction times and behaviors be evaluated. Other aspects that are examined in usability testing are as follows: 

• Is operation of the app largely self-ex­plana­to­ry? Are ex­pla­na­tions provided for more com­pli­cat­ed functions?

• Are app elements well placed, and important functions easily located by the user?

• Is the placement of elements and layout of the user interface organized in a way that is intuitive and easy to un­der­stand?

• Are the buttons optimally po­si­tioned and large enough to ac­com­mo­date users with wider fingers?

• Are texts clearly written and formatted in a way that’s easily readable and ap­pro­pri­ate­ly sized?

Usability testing should discover any problems with the operation of the app or short­com­ings that detract from its ease of use. In order for the results to be con­clu­sive, the usability test must be carried out by a suf­fi­cient number of test users. It is also ad­van­ta­geous if the test users (at least a majority) are people from the target group – that way the operation and structure of the app can be optimized for expected users. The resulting im­prove­ments allow the developer to offer the best possible user ex­pe­ri­ence.

The com­pat­i­bil­i­ty of an app in regard to the hardware and software of mobile devices should always be tested. In terms of hardware, one important example is testing the rendering of the ap­pli­ca­tion on different displays. Because there are different screen sizes and res­o­lu­tions, display and cus­tomiza­tion errors can occur rel­a­tive­ly quickly, es­pe­cial­ly if several layouts of the ap­pli­ca­tion for different screen formats have not already been created. In addition to the display format, CPU and RAM vary from device to device. Keep in mind that older mobile devices are not a rarity. Apps that are designed not only for current models, but also for older smart­phones and tablets, au­to­mat­i­cal­ly increase their market range.

In regard to device software, the smooth playback of an ap­pli­ca­tion is largely con­trolled by the different versions of the operating system – Android app testing on various versions of the operating system is es­pe­cial­ly important, as this platform has sig­nif­i­cant­ly more versions with more variation of use than Apple’s iOS platform.

Com­pat­i­bil­i­ty testing allows de­vel­op­ers to check which hardware or software processes cause immediate problems with rendition. If the re­quire­ments for the device com­po­nents are very high, it may be helpful to reduce certain system re­quire­ments in order to avoid shrinking the target group.

Per­for­mance tests show how many system resources an ap­pli­ca­tion consumes during intense use. Does the app continue to run smoothly, or does it ex­pe­ri­ence lapses? Beyond that, per­for­mance tests can determine if a device’s battery will become over­loaded or if the device itself will overheat after a certain period of use. As with com­pat­i­bil­i­ty testing, tests on the per­for­mance of the ap­pli­ca­tions are con­trolled under certain con­di­tions – for example, how the playback of the app changes if there is hardly any space left on the device’s hard drive or if the battery is almost empty. Per­for­mance testing is intended to reveal the extent to which certain cir­cum­stances neg­a­tive­ly impact the app’s per­for­mance.

If one or more com­po­nents are going to be subject to high uti­liza­tion, then a load test (or stress test) is used. One example of this is the server stress test, where the app’s behavior is tested in reaction high server demand. For ap­pli­ca­tions that require an internet con­nec­tion, a network load test is conducted to see how a weak internet con­nec­tion affects app func­tion­al­i­ty. Stress tests are designed to show how much stress certain com­po­nents can take before their per­for­mance becomes re­strict­ed.

The above tests don’t nec­es­sar­i­ly have to be carried out manually. Diverse testing tools can make the work con­sid­er­ably easier in many areas of app testing. Seeing as this article series discusses mobile ap­pli­ca­tion de­vel­op­ment for both Android and iOS, the following app testing tools are those which are con­sis­tent­ly available across both platforms.

Appium is a cross-platform open source program for test au­toma­tion that is available for download free of charge. This tool doesn't only test native Android and iOS apps, but also supports test au­toma­tion of hybrid apps and web apps. Appium uses the same pro­gram­ming interface for both iOS and Android, so it is easy to reuse pre­vi­ous­ly written code. The created tests can be carried out on physical devices, sim­u­la­tors, or emulators. Appium is in­cred­i­bly helpful software for the execution of automated app tests, however, the program does require the user to have a certain level of pre-existing know-how.

On TestFairy’s website, de­vel­op­ers can upload their app and invite people to par­tic­i­pate in app testing. These people can then download and use the ap­pli­ca­tion – es­sen­tial­ly, TestFairy allows for self-organized crowd testing. The tool has an error reporting feature and can record video of app test usage. TestFairy is available as a free download and also a paid en­ter­prise version.

Ubertesters is a platform that helps with beta testing for apps. De­vel­op­ers have complete control over the tests and receive detailed reports. On the platform it is easy to work together as a team, detect pro­gram­ming errors using various tools, create and test different versions of the app, and perform crowd tests (for example, in the form of func­tion­al­i­ty or usability tests). Ubertesters is offered in various packages, each with different features – the package with the fewest features is available for free download.

Obviously, the reach of an app can be extended by designing it to run on many older devices. The adap­ta­tion of an app to the largest possible number of devices, though, has its limits. CPU-intensive ap­pli­ca­tions, for example, whose functions require a stronger processor and/or more memory, can sometimes not be prepared optimally for older hardware. Another example: if an app requires a powerful internet con­nec­tion, it should ab­solute­ly not be adapted to any usage situation in which there is only a weak internet con­nec­tion.

Some ap­pli­ca­tions simply require certain technical pre­req­ui­sites. For example, some ap­pli­ca­tions require a par­tic­u­lar screen size in order to run. Another example is the YouTube app, which can only function with access to a suf­fi­cient­ly fast data transfer rate; yet it is still one of the most down­loaded ap­pli­ca­tions ever. Ad­just­ments and re­duc­tions of func­tion­al­i­ty in favor of a large app range should only be made if the app func­tion­al­i­ty is not reduced more than desired.