Long Term Maintainability

Once upon a time, you could design and build a device, flash a well-engineered firmware onto it and support it for 20 years, with minimal changes to the core software. Those days are over. A new approach is needed to provide long-term support for devices while managing the complexity of today’s embedded software stacks.

Programming has become harder in the last 25 years. Back in 1999, in the embedded software world, device firmwares were coded from scratch by small teams who could understand every codepath. Even the top-selling video game of ‘99 (RollerCoaster Tycoon) was coded by a single software developer working in x86 assembly language.

Today that is unthinkable. For all but the smallest microcontroller platforms, it’s no longer possible for a small team to understand every codepath. There’s no one person alive who understands all 30 million lines of code that make up the Linux kernel, never mind the additional software used to build, compile and deploy it. The techniques we use to provide “long term support” need to adapt accordingly.

Today’s software is developed by larger teams. In open source projects like Linux, the “team” is relatively informal and works largely in public. And in nearly all cases they are working on the latest versions of the project’s code. Nobody is volunteering much time to look at 10 or 20 year old versions of Linux. Those that do, are highly paid specialists, and even they can only do so much in terms of backporting fixes for today’s highly complex security issues, to kernels from the last decade.

That means, if you’re shipping a device with an old version of Linux, there are many codepaths on your device that nobody really understands any more. How are you going to provide real long-term support for a software stack that nobody fully understands?

Make your project Long Term Maintainable

To provide long-term support for a modern device, your software and your development processes must be Long Term Maintainable. This means being able to regularly update the software to modern versions which are actively maintained and understood by the development communities working on them. For devices running Linux, this means regularly updating to a modern version of Linux. As of 2023, that’s how you ensure the device is as secure and reliable as possible.

At Codethink, a core part of our business is helping customers to get to a point where they can fearlessly build, test and upgrade device firmwares. We’ve been researching and recommending best practices since 2008, when we first started helping customers with embedded device issues.

Achieving Long Term Maintainability is a matter of improving your development, testing and release processes. Common things we aim to polish:

• The build process must be fast, reproducible, and flexible. Your build inputs will probably change significantly during the lifetime of the device. (The ideal is to support distributed build).
• The test pipelines must be fully automated and cover every aspect of device functionality, so component upgrades can be done fearlessly.
• The device upgrade mechanism must be fast, atomic and capable of rolling back to older versions if needed.
• The team should proactively monitor security issues reported in upstream projects.
• The team should work together with upstream projects to test new versions of open source dependencies as they’re developed, and catch issues that affect your project before they hit stable releases. (See: "Why aligning with open source mainline is the way to go")
• All third party source code should be mirrored locally, so you can be confident the inputs to your build process will not disappear over time.

All of this can be automated as part of a well designed continuous integration pipeline, so project engineers are not wasting time on repetitive tasks.

What tools should you use?

We recommend using the tools that best fit your project. There are many great open source tools that can help you today. For example, we've demonstrated atomic upgrades on a phone using OSTree, and kernel testing using OpenQA and Lava.

That said, we do have preferences. In a few cases, where we wanted to try a fundamentally different approach to existing tools, we’ve built our own open source tools. Apache BuildStream is a great example where we saw a need for an integration tool that took repeatability and caching to the next level. BuildGrid is another example.

We've written extensively about Long Term Maintainability - here's a list of all historical posts about LTM. In the coming months we have several articles and exciting announcements. Follow us on LinkedIn, Mastodon or Twitter so you don't miss anything.

Contact Codethink, and find out how we can help ensure that your project is Long Term Maintainable.