Testing in a Box: Streamlining Embedded Systems Testing

Codethink has been developing a number of tools and processes in order to automate as much of the testing pipeline for embedded systems as possible. This includes everything from virtual hardware using QEMU, to testing on the hardware itself.

We have found that testing is often a neglected aspect of projects. It is often left to the late stages of development, leaving little time to fix problems that are found before a product goes to market. This in turn creates a stressful environment, causing “sticking plaster” fixes to be put in place rather than fixing issues at the root cause.

Testing is commonly a very manual process with teams of people scrolling through menus and performing actions on the device under test. This is both time consuming and expensive. It can also lead to the duplication of issues, unclear bug reporting and a lack of technical data to back the issue. By adding automated testing into the CI framework from the beginning of development, issues can be found and fixed early. The root cause is more easily identified, and fixed more efficiently, in a less stressful environment. This results in a more stable product with less technical debt.

We've used a number of different technologies on our journey including:

• Gitlab CI to run tests on daily builds or upon new merges
• openQA to run UI and graphics tests
• QAD to simulate keyboard, touch and gesture interactions
• CAN interfaces to simulate a vehicle on the road
• USB Switches to control multiple peripherals to the hardware under test

Typically, this has taken considerable time to set up. This can mean wading through the red tape of client's IT departments to gain access to client infrastructure, understand their build, CI and Test setup and integrate the tools required to provide useful data from testing.

Once set up and working, we've found that there are a significant number of dongles, adaptors, computers and cables hanging off the hardware under test. This increases the complexity and the chances of peripherals becoming accidentally disconnected, as well as looking untidy. The picture below can look familiar to some people reading this:

The fix

Whilst discussing a client project the conversation led to the question "Wouldn't it be great if we could have all of this in one box, easily configured to work with any hardware and without the need to access anything other than the hardware in question?"

So that is what we've done!

In order to tackle the issues of trailing cables and multiple peripherals, we have designed an IO board which facilitates serial I/O, I2c, SPI, GPIO, bluetooth/wifi, USB switching, a USB hub, Can bus emulation, HID emulation and even a host PC. As with the software, the box has been developed in a modular fashion, so you are able to only include the parts you need for a particular use case.

Also, we’ve our baseline software infrastructure defined in ansible scripts that we use/deploy in docker containers. This has been carried out with a modular approach, to give the flexibility to configure and deploy only what is required in a particular scenario.

To date, we’ve produced a first revision of the board, which had a number of issues which had to be rewired or otherwise resolved. There is a design for a Rev-B board which has been manufactured and tested. We are currently using this version in a variety of internal projects.

Future plans

There are plans for a version 2 board which will provide the same functionality but with less components that is even cheaper to produce! In addition, we also have plans to add a power distribution module to the box to allow us to control the power to multiple devices under test in a board farm scenario.

Currently, the tools to configure the hardware run as application specific CLI tools. We have plans to write an abstraction layer to bring these tools into a single user interface, which, will allow the user to configure the hardware from a single window.

Get in touch!

The project is entirely open source and is available here.

Documentation for the project is available here.

If you’re interested in learning more about this project and how we can help you, please reach out to us via sales@codethink.co.uk.