Tech notes: Android Studio Dolphin up the game for automation testing

Android Studio Dolphin(2021.3.1) is now available in the canary channel and will soon be moved to the Beta channel near the Google IO’22.

Android Studio Dolphin coupled with the latest version of the Android Gradle plugin(AGP) 7.3 introduces some really cool features to level up the game of automation testing especially around instrumentation testing in Android.

Some challenges in Android testing

Android testing is complex and becomes more complex when comes to instrumentation testing, simply because of ever-evolving APIs across Android versions.

As an Android developer, I really want to

1. Run my instrumentation tests across a range of different Android API devices/emulators.
2. Run-on different screen sizes.
3. I would love to run my test in parallel so that my build speed can be increased and I can spend less time browsing memes on the internet.
4. Screenshot test across API and screen sizes will make sure I am shipping the best possible version of my product out in the wild wild world.
5. Reset the state of emulators before every test.
6. and many more…

Good folks from Android Studio to rescue

With Dolphin and AGP(>7.3) now you can manage your virtual devices via Gradle directly. You can configure virtual devices from your project’s Gradle file and manage them from there itself. It supports the complete lifecycle of the virtual devices from creation, deploying the app and destroying the emulator once the job is done to save resources on your under-stressed machines.

As of now, the API ManagedVirtualDevice supports only four parameters

device: hardware profile of the virtual device

apiLevel: Android API level

systemImageSource: Source of the system image like “google”, “aosp”. As of now, it’s a bit weird and more details are expected in future as the API evolves and become more mature.

require64Bit: If you need to deploy 64-bit images on x86 machines. On arm machines 64 bit is selected and this field has no effect.

This will improve the quality of testing on Android virtual devices to great extent. It will use emulator snapshots to reduce device startup time and memory usage and use this to restore the device to a clean state before every test.

Test results will be cached and only those tests will be re-run in subsequent execution which are likely to give different results based on code changes.

A similar setup can be done on CI easily with server-side rendering support, it will ensure you have a similar test setup both on your local machine and CI.

To run your test execute the following command where “release-dev” is your build variant.

gradlew pixel3a_API32_ARM64_v8 release-dev

Group of devices

Similar to the above where we create a managed virtual device, we can scale our tests across multiple device configurations also by creating a device group. Gradle will execute your tests on the whole device group in parallel (see more optimisation ;) )

To run your test execute the following command where “release-dev” is your build variant.

gradlew phoneAndTablet release-dev

Automated Test Devices(ATD)

A new emulator image is also made available(at the time of writing only in the Canary channel) to further enhance the performance of instrumentation testing with optimisation across the android AOSP.

Nothing special is required to create an ATD, just select the “atd” image while setting up your virtual device.
(At time of writing ATD images are available in Canary channel only)

A lot of unnecessary apps are removed from ATD images to save the memory footprint of emulators/test devices plus give a boost to increase in performance of the device. The decision is based on unit testing principles that your test should only be concerned about your app and other Google apps(like Gmail etc) or service should be out of the scope of your app’s test. As ATDs will operate in headless mode as most of the services are pruned from the images.

Google Apps like Maps, Mail, Chrome, Messages, Play Store, Settings, Telephony, System UI, Browser2, Calender2, Camera2, Contacts and a few more are removed from ATDs. As the documentation clearly states that these apps and services are outside the scope of automated tests for your app. You should use intent-matchers to validate the outgoing intent in your tests to determine success and failure.

Settings-Provider will be part of ATD images.

Test Sharding

If you ever worked in a huge app with thousands of test cases you must know how much time it literally takes to complete the final builds. A lot of organisations come up with their own solutions for this kind of problem. Time taken to run all the test cases is a major part of your build time apart from Gradle or those annotation processors’ sorcery.

Now AGP supports test sharding, which means you can run your test suite in parallel on multiple test devices called shards. You can set the number of shards in your gradle.properties file.

android.experimental.androidTest.numManagedDeviceShards=3

Gradle will provision the number of shards you specified in the properties file for each managed device in your Gradle file. So be conscious about the resources it can consume while creating shards. So for example, if you have 3 managed devices in your gradle, and you mentioned 3 shards, Gradle will provision 9 virtual devices to run your test suite.

Your test results will be outputted as .proto file for each shard used.

Sharding can be very resource-intensive, so be very conscious while setting up your test setup. If Gradle is unable to provision a test device due to a resource crunch it will result in test failure with time out error.

Caveats

This world isn’t fair…- Randy Newman

1. As you must have learned to use all of the above, you have to bump up the Android Gradle Plugin to 7.3 or greater.
2. You need to use Java 11 to use AGP 7.3 or greater.
3. These features are available only for android images from API 27.
4. Screenshot tests are not supported as now with ATD images. :(
5. The API is still experimental and subject to change.