Layering it on--a new approach to automating system tests

When building software, you always need to know your software's quality. Testing is the obvious vital task in assessing the quality of a product, but it's also a major contributor to product development time. Automating tests can shorten the time your spend on executing test sessions, while improving resource use and extending the number of functions you can validate.

This article presents the pitfalls and challenges posed by creating an automated test strategy for an embedded system, namely a Voice over IP (VoIP) media gateway. The layered approach we devised doesn't necessarily reduce testing effort but instead converts tasks, such as execution, validation, monitoring, and reporting, into software routines.

A layered approach with pluggable components assures scalability and portability, enabling you to test a class of products individualized by the capabilities and hardware platforms they run on. Although the diversity in embedded systems is so great that no one generalized testing method can be applied to all different engineering areas, let alone among products from the same specific area, we believe our basic concept can be successfully adapted to other classes of embedded systems.

Our system under test (SUT)--the VoIP media gateway--consists of a complex integration of software modules and control running on a specialized processor optimized for both signal and packet processing. A common requirement of embedded systems is the ability to process data in real time. A VoIP media gateway processes voice signals within a specified time period, which must be as short as possible to reduce the overall communication delay introduced by packet networks and processing nodes. The real-time environment makes the SUT's behavior more difficult to predict and analyze. Defect occurrence in real-time systems is sometimes nondeterministic, as it becomes visible only when special conditions are met.

We validated the media gateway from a system perspective--we considered all the components to have been unit-tested prior to integration. The output of testing this SUT consisted of media and control packets, digital signals, and debugging/logging information. We based our test approach on standards, reference or relative data, and interoperability with existing equipment.

Why automate testing?
The numerous stories of failed attempts at test automation all begin with an ill-conceived objective for the project. While figuring how much time and money you will save, note that for the most part, test automation doesn't leave testers with more free time on their hands. On the contrary, automating a testing project is a full-time effort, not a sideline task.

The central question is whether your objective is to save testers' time and your company's money during testing phases or rather to end up with a higher quality product that has fewer chances to fail during its service life.

Starting an automated testing project is actually a lot more work than purely a manual effort. Test automation doesn't replace testers, rather it changes their focus. People are needed to program the scripts, set up the test runs, interpret the results, discuss fixes, and so forth. All these make test execution actually a small part of the whole testing effort. Testers typically don't end up with less work to do.

So, why then would you want to automate your testing? See the sidebar below for advantages and disadvantages of test automation; or go to next page to continue the article.