How to Choose the Right Devices for Mobile Testing?
On top of all this, there are thousands of mobile devices that are already in use. How do we avoid problems with multiplying application errors on different devices? How should we deal with the selection of devices for existing and new projects? I will try to answer the questions below.
Why is it so important to pay attention to the selection of test devices?
The Statista report shows that in 2022 the total number of devices globally will reach 16 billion. Year to year, it is estimated that this number will increase by another billion. Of course, we don't have to worry about all of them (but more on that later.) This number only shows the tremendous challenge development teams are facing while building applications.
Let's assume that the mobile application has been tested on just a few devices. All the testing devices were manufactured in similar years, have similar versions of the operating system, and have similar screen sizes. The idea of the app is excellent, the application looks and works great, it has been tested, so we are ready to go a step further – we are releasing the application to stores and waiting for user feedback.
After a short time, the ratings in the stores are terrible, there are complaints about the interface, performance, and user experience, and some users were not even able to run the application.
The idea of our application has quickly became invalid. Multiplying problems have effectively discouraged users from using it, and in-store reviews, ratings, and comments drive users away from downloading and installing. Users switch to the competition very quickly. The development team fails.
This scenario sounds terrible. Fortunately, this is only a hypothetical situation that can be avoided by a conscious process of selecting test devices before starting the application development process.
What should be considered in the process of selecting test devices?
I should start with whether our application is still in the early development stage or is already in use. If the application is being used, we can search for the available data in Google Analytics – a precise method of finding online user data (if it is in use) or different tools such as SimilarWeb, etc.
If the application development process is just beginning, use the statistics available on reliable websites such as StatCounter.
Who is your potential audience?
The most crucial thing will be which group of users we target for our application. Pay attention to the fact that some users may be more technologically advanced, therefore more unconventional solutions can be prepared for them. Users who, for some reason, are not technologically advanced and cannot operate smartphones smoothly will need simple solutions.
When defining the potential audience, we should recognize which group of people our application is targeting. When selecting the target group, you can use diversity factors such as age, gender, language, location, interests, education, or even the tendency of users to update the application frequently.
Sometimes the applications we create are intended for internal use. In these cases, the process of selecting the correct test devices may be similar. As the development team already has an end-user knowledge base, we can base the choice of devices on statistics for the most popular devices, operating systems, and screen sizes, taking the target region into account.
What is your application’s destination region?
Each country/region has its own trends in the use of mobile devices and OS versions. This is due to several factors such as
- Local manufacturers
- Availability of foreign devices
- User wealth/cost of devices
- Language
- Level of education
Thanks to the StatCounter website, we can collect accurate information on the most-used mobile devices and operating systems globally and per region. Remember to specify the last oldest supported version of the system. It is not possible to support all available OS versions. Therefore, after prior identification of the audience and the region, we are able to determine the minimum supported version of the device system.
Who are the leading device manufacturers?
Tools such as StatCounter collect and frequently update data on the most popular brands of mobile devices. Thanks to these statistics, we can find out which manufacturer is the most popular in a given country/region.
What are the most common screen sizes?
When choosing mobile devices and screen sizes, we should also consider the target groups and their features. The selection factor may be an aspect such as
- Age
- Location
- Availability of particular devices in a given location
- Technological advancement of potential users
As in the case of the most-chosen manufacturers, devices, and operating systems, the sizes of screens can be determined on the basis of statistics provided by websites such as BrowserStack. Having information about the most popular devices, we can determine the supported screen sizes and resolution.
What number of devices should be tested?
When proposing the number of selection of devices for testing, I base this on analyses carried out by BrowserStack in 2020.
BrowserStack looked at global trends in the use of mobile devices and the popularity of devices. The sample for analysis was over 2 million users. After the analysis, BrowserStack was able to determine the combination of devices that gave the maximum market coverage.
As a result of the analysis, BrowserStack identified three categories of device recommendations based on website traffic. The conclusions were as follows:
- Up to 15,000 unique users per month – 6 devices covered 93% of the iOS market and 76% of the Android market
- 15,000 to 300,000 unique users per month – 14 devices covered 93% of the iOS market and 89% of the Android market
- Over 300,000 unique users per month – 24 devices covered 93% of the iOS market and 97% of the Android market
The essential thing is that effective testing is not about selecting as many devices as possible, but about the most accurate combination of devices/operating systems and screen sizes.
Ideally, all test devices should be physical, but we are not always able to operate all the defined devices at a given time. We can choose a smaller number of devices for manual tests than for automated tests due to the availability of devices and greater time consumption of testing.
This is where BrowserStack solutions come in handy, where you can easily use real devices, (but not physical devices) that are only available in the browser.
I will not suggest one specific number of devices for testing because it requires a detailed analysis of the aspects that I write about in this article.
Improving the software development process
Knowing what we already know about the selection of devices for testing, we can improve the application development process from the very beginning, even at the stage of collecting requirements.
The awareness of the systems and devices on which the application is to run will allow the team to choose the right tools for the distribution of the application. It provides the comfort of not only QA, who then know which devices to test, but also the comfort of the developer, who knows from the beginning which devices, systems, and resolutions are supported, which will avoid any doubts.
Costs and challenges
It can be said that providing QA for all test equipment can be expensive. This is true, and it is a challenge. It should be remembered, however, that the competition is awake and users are more and more demanding and even a slight inconvenience may cause the application to be uninstalled. About 25% of users use the application only once after installation. Localytics shows that 50.57% of users churned one month after an app was downloaded.
If we want to ensure that the application works flawlessly, it is worth investing in testing devices.
Another important aspect is the elimination of costs that may potentially occur due to insufficient selection of devices during the tests. The better the combination of devices we use, the less likely end users will encounter errors.
