Automated end-to-end testing for Android Automotive on Hardware

Codethink has automated some functional testing of Android Automotive on our public openQA instance. These tests run on a Raspberry Pi 4 and we intend to automate the build and flash process, so that we have the possibility to add them to CI pipelines on any given project. See an example test run here.

Why does this testing matter?

At Codethink, over the past few years we have been focusing on how we can improve the state-of-the-art for testing. We’re big fans of continuous integration, where we want to constantly integrate and test the latest version of every component together at the same time, in order to catch issues or conflicts as early as possible, when they are less expensive to fix. But, whilst constant build/compile testing is incredibly useful on its own, when it comes to software testing, CI pipelines will all too often only run unit tests, and teams overlook the need for functional testing that verifies the whole system, where we replicate both the end-user environment and also how the user will actually interact with the software. We call this end-to-end testing.

Currently, to achieve the level of confidence required, many large projects are reliant on manual testing, with some having hundreds - even thousands - of testers. Whilst some manual testing will always be needed, we really want to be able to automate as much as possible, making more efficient use of the hardware resources (overnight, for example) and freeing up human beings to do more interesting and more valuable testing.

So we are not trying to replace unit testing: we are trying to replace manual end-to-end testing as much as we can, because we believe projects should be performing robust, valuable testing as early as possible in the development process.

To achieve this, we turned to openQA, an open source tool developed by SUSE, originally designed for end-to-end testing of desktop Linux distributions. We've built a number end-to-end test pipelines since then, some for our customers in the embedded and automotive world, and some public pipelines that provide extra testing for open source projects. We're continually testing mainline Linux on a set of development boards, using LAVA and openQA; we spoke about this at FOSDEM 2022. We developed QAD, a lightweight alternative to openQA's usual remote desktop backend, and an open-hardware USB Switch for automating tests that require physically connecting and disconnecting USB devices from a test rig, and we spoke at EOSS Prague 2023 about this work.

We’ve now added Android Automotive testing, using Android 13 from Snapp Automotive and a Raspberry Pi 4. This matters because a lot of our customers are turning to Android Automotive, and we want to be able to test that in CI from day one of the project, and easily add additional tests over time.

Next Steps

The tests are currently quite straightforward, and include booting to the home screen, navigating to the apps screen and then back to home. In the near future we want to add tests for CarPlay screen projection and USB Media devices.

We are also looking to automate the build and flashing process (currently manual), in order to allow us to frequently update to the newest version of Android Automotive. With this and the additional tests in place, we can usefully start to find and report bugs against the Snapp Automotive project.

Interested in learning more? Contact us at sales@codethink.co.uk.