What are Device Farms
Device Farms are services that provide cloud access to devices (smartphones, tablets, smartwatches) and desktop web browsers. Thanks to remote access to real devices (not emulators) we are able to run automated tests on them or manually click through the tested application on a selected device, thanks to which we have coverage of a large number of real environments. Some producers make devices available in the farm before they are released, so this is the best way to test applications on these devices before they are released.
Why Device Farms were created
Device Farm addresses the problems of legacy testing solutions on emulators or physical devices.
Emulators do not always reflect the problems found on real devices. Emulators also have limitations related to device location settings, interaction with other applications such as pop-ups from other applications. Therefore, testers report bugs that occur on devices and developers often fail to reproduce them on the emulator. Emulators do not provide information about CPU usage or battery consumption.
Testing on physical devices also has its limitations. While large companies are able to provide many different devices, for smaller companies it is a big financial burden. Therefore, the number of devices is limited, which negatively affects the coverage of environments during testing. Additionally, remote work complicates the exchange of devices between employees. Owning devices by a company means the necessity to maintain them and constantly replenish them with new, popular devices. It is also problematic to receive error reports from customers on untypical devices, where the error is suspected of a specific device (device specific).
Device Farm also allows you to test web browsers. Running multiple automatic tests on a computer we are limited by the technical parameters of our device. The number of sessions and versions of browsers is limited.
How it works
After creating an account and logging into the service, we choose devices or browsers on which we want to run the tests. Then we attach the mobile application to be installed on the devices.
In case of testing desktop browsers, device farm allows to run several dozen sessions simultaneously on different browsers in different versions, which significantly shortens the time of performing tests.
In tests on mobile devices we choose devices and versions of operating systems on which we want to run automatic tests. The tests are performed simultaneously, thanks to which the wait for results is relatively short. The obtained results show which devices passed the tests successfully and on which devices the tests failed. Some Device Farm platforms provide logs, screenshots, videos, information about processor load or battery consumption.
You can also select a device to manually click on. As the devices are real devices, you can install other applications or enter device settings if the tests require it (device language, date/time, location, Bluetooth, Wifi, etc.).
Services offer plugins and APIs to automatically initiate tests and receive results from IDEs and continuous integration environments such as Android Studio or Jenkins.
– Device Farm allows testing in a real customer environment
– Large selection of devices (the most popular services have about 400 models of devices)
– Access to devices before their introduction to the market (e.g. Samsung)
– No need to buy and maintain devices (lower input costs)
– Large scaleability of tests (large coverage of environments)
– Access to devices regardless of location
– maintenance costs (periodic access fee)
– higher risk of security breach compared to local checking on the emulator
Over the past few years, Device Farm services have increased the range of devices available and this trend is sure to continue. The services also offer increasingly powerful reports and connectivity to other test and code management software (e.g. Android Studio, Jenkins).
Recently, the first services were created that offer building device farms inside the company, which will increase the security of use, minimize maintenance costs and facilitate access to devices inside the company, which in times of popularity of home office is extremely important.
Device farm is an interesting solution to support tester and developer work. Increasing the coverage of tested devices in the real environment (and not only in the emulator) will positively affect the quality of released applications. The ability to run automated tests on multiple environments at the same time shortens test execution time, while the option to manually test any device is especially important in times of widespread remote work. It is worth watching the development of such services because they may become a basic tool used in software development.