Mobile Testing Essentials

I wrote several articles about my overall approach for testing. I base all things I wrote on my personal experience, mostly in services companies. And this fits for small teams under budget. In a big corporation, you probably have your own rules on how to test. So if you do something (or everything) completely differently, it's totally fine. I hope these articles help QA engineers to start working on a new project and cover the most important things.

Today, we'll speak about mobile projects.

Step 0. What you better to do before starting any testing activities

First of all, you need to know what your team, managers and customer/product owner are waiting from QA engineer. So is better to ask about their expectation in your first days (also it's a good question to ask during your job interview). Approximate questions:

• what they exactly wait from QA: reports, bugs, test cases, other forms of reporting (like everyday email etc);
• how the testing process looked before you join the project, if no — what would be the ideal. This question is a little tricky — you don't need to do everything what your manager tells. It's just useful knowledge for you.
• what the developer team is waiting from QA. In general people are talking with managers, but in real life most of the work time we will communicate exactly with dev. team, so it's good to know what they are expecting.

You can always ask something more, everything what is important for you. Answers and all the information gathered during this session help you create two basic test artifacts: Test Plan and Test Coverage. There are a lot of templates on the Internet, but I suggest using templates from ISTQB. They are available for free and many people at least hear about them.

Step 1. How to choose right devices for testing mobile apps

To estimate testing time the one of important things is to know how many devices there are under test. In this section I'll show how I choose devices for iOS and Android, all these could be appliable for other OS and types of devices (tablets and smart watches for example).

1.1. Choose OS versions

First of all, you need to find info about target OS versions. In general, there is a requirement like " An app should work with iOS 10.0 and above and Android Oreo 8.0 and above)".

If there are no requirements, the better way is to choose two last available iOS version. It works because there is a high adoption rate for newer OS among apple users, so you always need to check how your apps work after major releases.

For Android it's a little complicated. You can choose target countries and check what are the most popular OS versions in there. If you don't have target country, you can choose the world statistic. But in this case, you should be ready to update the target OS when you'll receive more data about your audience.

1.2. Choose right models for iOS

Now it's time to create your own park of devices. You need to reach maximum coverage of screen resolutions and popular OS versions. If you can check your app on different screens, you can find the most popular bugs.

To choose iOS devices, create a table for all available and divide them on groups by screen resolution:

From this table you can see that there is no reason to test UI on iPhone 12 and 12 Pro: they have the same resolution and UI will be pretty similar. Also good to know what @2x and @3x mean, I took Daring Fireball’s explanation:

@2x means the same “double” retina resolution that we’ve seen on all iOS devices with retina displays to date, where each virtual point in the user interface is represented by two physical pixels on the display in each dimension, horizontal and vertical. @3x means a new “triple” retina resolution, where each user interface point is represented by three display pixels. A single @2x point is a 2 × 2 square of 4 pixels; an @3x point is a 3 × 3 square of 9 pixels.

1.3. Choose right models for Android

For Android devices is a much complicated thing. But the main approach is the same: collect info about the most popular devices in the region or worldwide, divide them by screen resolution and choose the popular one from every group. Here is an example of most popular Android devices in Poland:

Data from https://www.appbrain.com/

I would like to add here my thoughts about minimum amount of needed devices. If you're working in a small team with 1 QA, it's better to limit your devices up to 6 (3 Android and 3 iOS). Testing time is limited, and it's fine to test major design updates on every screen resolution. But during sprints it's enough to test only on 2–3 devices (depends on how many hours do you have).

Step 2. Prepare right tools for testing

As soon as you created your list of devices, it’s time to think about which tools do you need. Here I gathered my favorite tools that help to cover basic needs for every mobile project.

To test the app, we need to install it first. So adb is a perfect helper here. This is a command-line tool for any android device. It means that you can use it not only with mobile apps but for every device based on Android. There is no user interface and all you can to do is write some commands and read answers. The common way to use it is:

• installing/uninstalling/running/stopping apps,
• receive logs in real time and check crashlogs after crash happened.
• push/pull files directly to a device.
• The list of all available commands you can receive by "adb help" command or check them here.

Unfortunately, there is no so handy tool for iOS devices. To see logs from iOS device in real time, you need to build the app inside xCode on your Mac machine. This is a rare case when QA Engineer has the mac machine and access to source code. So one of workarounds is to ask the developer to save logs from the device inside the app. Then you can perform some testing, find these logs on the device and check for errors. If you don't have access to logs, the easiest way for testing iOS apps is just to check UI and database content directly. Also, you can find some crash reports in analytic tools (Firebase and similar).

What the logs are?

Logs are messages that your app is recorded while you perform different actions inside the app.

Why do you need to check logs?

They can help you investigate what steps have you done before the crash/errors happened. For the most cases, crash report contains the last string of code what has been processed before crash happened. It helps developers to fix it much faster. That's why logs are essential for crash investigating. Unfortunately, sometimes crashes are not contained any vital information and only proper steps to reproduce will help in the bug fixing.

The next tools I suggest to use are Charles and Postman. Both guys help you send and receive data between frontend and backend. Below are several examples of what we can check with it:

• UI is ready but backend is still in development state: while error responses are not ready, change with Charles the normal response to incorrect one and check how errors are displayed in the app.
• You need to check rare responses from the server and you don't know how to reproduce them: with Charles change any response to needed, check UI.
• You need to check how different data displayed in the app: again change the response with Charles.
• You need to check how different data sent from the app is stored in database: use Postman and send one request with different data.
• You need to create a lot of entries and check how the app works: use Postman, send one request many times. Alternative way: create all needed entries right inside the DB with DB tools.

Tools for databases are different and depend on a project. So no need to remember all of them have pretty similar interface and contain:

• tools for showing tables and data inside, tools for writing requests, creating/editing tables (but this is rarely available for QAs).

I mostly use them just to check if data is recorded and sent correctly.

Another useful group of tools you’ll use is distributing tools. End users receive the app using app stores. But you, as a tester, receive it in another way. For iOS you need to register all your devices in an app like Appcenter. Then you’ll be able to receive updates directly to your test devices. Also, these apps help to gather more data (logs, user actions) then released versions. In iOS development, the most popular is Testflight. Release candidate is often distributed directly with Testflight, as it simplifies future app release. For Android apps, everything is the same except Testflight. But there is one more way to get the app: a developer can send an .apk file on any messenger. But it makes version control difficult and should be used only for exceptional cases.

These are my basic tools for testing mobile apps: Charles and Postman help to check both frontend and backend parts. Any available tool for database check. xCode and adb for debugging.

Step 3. What you need to remember when you test mobile apps

When you prepare a test plan for mobile apps, you need to pay attention to the next topics:

• Publishing requirements. Apple has strong requirements for the apps, starting from design and ending with anti-fishing practice. Especially, you need to pay attention to all requirements if you create children's apps.
• Subscription. Most of the apps have different subscriptions. And it’s essential to test them before release.
• Permissions. Very important to check how the app works if user declines permissions. Also, there could be some functions inside the app with special requirements. And if developer forgets to set them, the app will be crashed. Examples: inside the app user can take a photo. For example, to add a note with text. But when the app was started, there wasn’t request for camera permissions. And without them, the app just crashed.
• Work with external API. Here you can find a lot of bugs and crashes if app works with external analytics tools, external API and any other integration tools. Example: in your app for buying tickets, user can check the weather. But the weather is requested from external API. And they have today bad day and didn’t send needed data for your app (or send it in a new format). Here you can just not seeing needed data or app will be crashed on start. Or on this page.
• Accessibility. You need to be ready that your app is used by different people with different needs. So don't forget to test its accessibility settings and sometimes it worth to check with Zoom mode on iOS and Magnification on Android.

I hope this article helps you to work on a new project easily. In conclusion, I want to say that there is no magic advice that fits every project. Also, everything changes so fast so don't be aware to use and implement something new in your project. Good luck!