An interview with James Grenning

4/5/2010 9:44 AM EDT

With respect to the guest, I think he is an example of the kind of software person who focuses too much on the software itself and not enough on the whole system. Consider:

> The code specifies the behavior of the executable program in all its necessary detail

That's a true statement, but a misleading one. Only the target processor can read the code to the level of precision needed to answer key questions, like, "What is the maximum travel of this actuator?" or, "what is the maximum RPM of this motor?" Therefore, human reading of source code is no substitute for a concise, accurate, top level specification, which feeds both implementation and system test activities.

Here's an example: Some years ago, I heard about a software related incident at an automated sawmill. The target processor was 16 bits, and the software was calibrated in .001 inch units. The first log that came through the system with diameter greater than 32.767 inches caused a reversal of some actuator that broke stuff and injured the operator. So code may equal design, but in this case something more explicit was needed. I daresay that TDD-style unit testing would not have caught that problem either.

So one difference between embedded and other software is that physical systems often have complex requirements of their own. Those requirements need to be captured independently of the code, and need system-level testing independent of any in-code specification.

I think a certain personality type likes to pick apart the waterfall method, and there are obvious arguments against it. But so far, it is the best tool going for managing complexity and risk in real world projects. You can use TDD or OOD or whatever you need to use inside your waterfall block, but your block has to be correct in relation to the other blocks of the project.

4/5/2010 12:40 PM EDT

TDD is a way to ensure that some analysis and design is being addressed "up-front", but this article didn't cover all my concerns.
I've seen a lot of projects run into trouble, when integration occurs, due to a lack of planning for integration of the units. TDD doesn't seem to address this.
I also believe it's very beneficial to have independence in testing, even at the unit test level. It is too common for developers to write tests to prove their code correct rather than to prove it incorrect. Under time pressure, this becomes a subconscious driver. It would be really easy to write "soft" tests in TDD as well. Addressing this would probably result in developers complaining about BTUF (Big Testing Up Front).

4/6/2010 10:36 AM EDT

I think we all agree that the product must be tested as whole in its intended environment. Or...? The requirments may be flawed or incomplete, numerous research have labelled poor requirements as the most common cause for disaster.

Also, no matter how loose coupling your code has, there might be unforseen dependencies arising when the code is put together. All modules in your project will also share the same system resources. And there will always be the top-down dependancy from main thread -> code modules -> objects/functions. So the code too needs to be tested in its intended software environment, just like the final product needs to be tested in the intended "real world" environment.

None of the above excludes "TDD". But of course TDD can't be used as a replacement for final verification/validation, that would just be plain stupid, for the above mentioned reasons. I really don't hope that's the case...


> Here's an example: Some years ago, I heard about a software related incident at an automated sawmill. The target processor was 16 bits, and the software was calibrated in .001 inch units. The first log that came through the system with diameter greater than 32.767 inches caused a reversal of some actuator that broke stuff and injured the operator. So code may equal design, but in this case something more explicit was needed. I daresay that TDD-style unit testing would not have caught that problem either.

I'm going to be mean and shoot down that example.

Any decent static analyzer (and good compilers as well) would have attacked that particular bug saying something like "implicit conversion between signed and unsigned int". If you scratch your head for a moment, you will realize that the tool is telling you: "why are you using a signed int to store a diameter for? It doesn't make sense." You will then no doubt start digging in that code piece, finding numerous other bugs.

So I daresay that any form of test would have found that bug in a few minutes, if it involved a good compiler or static analyzer. As I see it, the bug was likely caused by any combination of the following:

- A poor requirements specification. Did the specification state how thick logs that were allowed? If it did, was the product tested against that requirment?

- Poor test tools or no test tools at all.

- Insufficient knowledge in embedded programming. A qualified guess is that the program was written by an unskilled programmer who was using the default "int" type - a deadly sin in any embedded programming. But in that case the real culprit was poor coding standards at the company, or no coding standards at all.

4/6/2010 5:26 PM EDT

I am afraid I have to disagree with all of you. My experience in embedded software supports every word Mr. Grenning said. First in my 30 years in the field I remember only one brand new first release program that was a total mess, and I’m sure that was deliberate obsifucation. Most developers seem to be able to divide a program into reasonable modules, and implement these modules intelligently. This is true no mater what process the original developers followed. It is enhancements that destroy the programs. This was true when new breath types were added to a medical ventilator, new measurements to optical inspection systems, or new protocols to telemetry systems. These changes are almost always made under time pressure, and use as much existing code as possible. The result is usually that pretty decent modules grow to be un-testable and un-maintainable. I have had to take over far too much such legacy systems, and it is very hard to convince managers that it is costing them more than it is worth, If TDD can stop this code rot and force needed refactoring I hope every organization whose code I ever have to maintain will adopt it.

P.S. Kent Beck may have taken the, write a little code, test it and extend it, method past any reasonable point, but that is how real working programs are developed whatever the official process may be.

4/7/2010 3:18 PM EDT

Mr. Martin, you don't know my background, so its odd you are making assumptions about me after a five minute read.

To clarify what you think is misleading, tests become the detailed description of what the code is doing. That does not mean that there are no specifications of what is needed. It also does not mean that no design documentation is needed. My point in referencing Mr. Reeves' paper is to give code its due respect. It's the only place where the detail has to be right, or digitally controlled saw mills start hurting people.

As I wrote in my answer to Jack's question about design, design specs are different in an Agile/TDD environment than in a design up-front approach. They should be kept high level, leaving the details to the code and tests. They should also be created economically, which probably means not spending so much time up front guessing about what might work before the design ideas have been tried and proven. Spend some time up-front, but beware of diminishing returns on your effort. Have enough to unify the direction of the team. Document as you go. Documenting what works is easier than predicting what works, and it won't need to be changed so much.

I'm curious, was there a top-level specification for the sawmill? What was the largest log diameter the system is designed to handle? What were the A/D limitations? What was the system supposed to do when a log that is too big is fed into the system? The are requirements that need to be addressed.

If requirements are poor, we get the age old problem of garbage-in garbage-out. When the people doing the programming don't understand what the code is supposed to do, how can they get it right?

Up-front, we never know in total what the code has to do because we are learning and discovering during the whole development process. This log width spec sounds knowable. Why did it fail? Not enough up-front documentation, wrong specs or sloppy/buggy code? I suspect the later, but I could be wrong.

Assuming that the sawmill requirements spelled out the system bounds, the critical defect of the upper bound edge condition is exactly what a thorough set of tests is likely to eliminate before ever mating the software with the hardware. If it was not in the spec, the developer using TDD would likely ask for the boundary behavior as the method encourages boundary identification and testing at the boundaries.

Finding that critical defect on the first log is the kind of problem that is going to continue to plague engineers that insist that the only place to test embedded software is on the target hardware. It needs to be tested on the target, but not only on the target. If we don't have comprehensive automated test suites, bugs will continue to infest our code. TDD does not eliminate all defects, but it does eliminate many more than any other method I have seen, and I am not new to this work.

I actually did not say much at all about waterfall, except that people that practice it, actually don't. They do something that works. It might look like waterfall on paper, or there are phase gates, but in the trenches people iterate and learn. Waterfall is too simplistic and does not account for feedback and evolution of goals and needs. I'm not saying specs are not useful. I say it is naive to think you can finish the requirements spec before development except in the simplest systems. This is not only my opinion, but the opinions of hundreds of engineers I have polled about it over the last 10 years.

Readers might want to look at Craig Larman's research on the topic published in IEEE Computer: Iterative and Incremental Development: A Brief History (http://tinyurl.com/ydl8eyt). He notes that IID has been around a long time, and used to deliver multi-million line critical systems like the Light Airborne Multipurpose System which is part of the Navy's helicopter-to-ship weapons system, as well as others.

Iterative is realistic when learning and discovery are part of everyday work. We learn; assumptions change; designs go bad; all resulting in code changes. Iterative development is realistic in this environment and TDD supports iterative with its regression test suite.

4/7/2010 3:52 PM EDT

Hey Chief Frog

Back in the old days the big problem was the late stage integration. It still is a big problem for many teams. But some are making progress in this area. In an Agile development effort, work is broken into cross-component scenarios or stories of limited scope. If you are familiar with use cases, a story is similar to a single path through a use case. The stories cut across the layers of the system. Each iteration (2-4 weeks) results in integrated and demonstrable code, maybe even releasable.

A companion practice to TDD is Continuous Integration. Several times a day engineers integrate their code, preferring regular small integrations over one big-bang integration. With stories and CI late stage integrations are avoided as much as possible.

Before getting into independent testing, the tests written while doing TDD are tests that assure the programmer that their code does what they think it does. These are fine grained tests written in lock step with the production code. In Agile development programmers consider it their professional responsibility to unit test their code. Some agile teams employ pair programming or frequent test and code reviews as a way to get more eyes on the work. That might get you the outcome you are looking for with independence at the unit level.

TDD at the unit level is only one of the Agile test practices. There is acceptance testing and system testing as well which I think would meet goal of independent testing.

The details behind the stories are defined by the product owner and qualified test people in the form of acceptance test criteria. The goal is to automate all the test criteria, but *all* is rather impractical with some of the interfaces found in embedded systems.

Instead of producing a big book of manual test procedures, what if you had executable use cases and a smaller book. These test cases are not something that is only done at the end of the development effort, but continuously throughout the development cycle. Manual testing would also be done throughout the cycle. This sounds like a lot of effort, and it is, but the effort might mean that some other activity, like writing use cases in natural language, won't be needed.

4/7/2010 4:01 PM EDT

Mr Ludin

Yes the code must be tested in its intended environment, but not only in its intended environment. As you can read in my prior post, testing happens at the end, but not only at the end. When testing happens only at the end, expect surprises, surprises that will destroy your carefully planned schedule.

To build reliable systems we need it all: good requirements, static analysis, tests and above all good skilled people making it work.

4/7/2010 4:18 PM EDT

Mr The Heretic

How did that 1722 line function get that way? Did the original designers say, this is the perfect place for a multi-thousand line function? No! Then how did it get that way?

One line at a time.

It's our professional duty to keep the code running, and that means automated unit tests at least. It's also our professional duty to keep the code clean so it is maintainable. We refactor to keep the code clean. Tests give the courage and safety net to refactor.

Thanks for the dialog all!

4/8/2010 2:05 PM EDT

Mr. Grenning -

Thanks for taking time to reply to my post. I did not really mean anything personal, and apologize for my careless wording.

The sawmill project was not mine, I just read about it, so I don't know if they used LINT or even if it was coded in C.

My only personal experience with TDD is a pilot that went bad. I have written about this pilot before in comments to Mr. Ganssle's excellent column.

In this failed TDD pilot, the unit tests generated were mostly trivial, and tested for trivial things that most programmers get right anyway. The tests also reinforced some misreadings of the project requirements and interfaces, making rework more expensive than it should have been. The code was "good," i.e., self consistent, testable, lint-free, whatever. But it did not match the real world around it. In my judgement, in that case, the time put into unit test maintenance would have been far better spent on interface validation. That's not just hindsight, it's how all my other projects at that company were successfully managed.

I do see the value of a good unit test in lowering the risk of refactoring as features are inevitably added. But I have not seen the advantage of paying for a comprehensive unit testing program, over "just" writing a good unit test for the function you need to refactor, when you need to do it.

Does that make me a heretic or just a dinosaur?

Thanks,
Larry

4/8/2010 2:24 PM EDT

James,

Being somewhat experienced in the field, I must say that the true benefit of experience is realizing that we must keep evolving. Quoting yourself when you wrote _ "engineers that insist that the only place to test embedded software is on the target hardware" _ they are holding themselves to a time where challenges were totally different, tools were much simpler, and the plethora of resources today available and for free at any desktop simply did not exist.

To discuss the agile manifesto in-depth would take days, but TDD clearly is a practice which only aggregates to our field. And it may be done either on the target or not, and preferrably at both, and also with some beautiful physical automation if the bugdet allows it.

It is important to emphasize that you have never advocated not to test it at the final target, as it would be just irresponsible.

I shall provide my testemony here that after adopting some TDD ideas, and investing a lot of effort on creating proper abstraction layers to test embedded code at the desktop, I observed my productivity and the confidence on my code just increasing more than ever. It even made me a better programmer, as I had to pay even more attention to differences in type sizes, endianness, etc, as any responsible professional on the embedded field ought to do.

4/8/2010 7:19 PM EDT

I'm a complete fan of automated testing, but I think it is perhaps a stretch to talk of test driven design (ie that the testing generates the design).

What sometimes shocks me is how little embedded software engineers make use of tools that regular software engineers use on a routine basis. Even tools like source control/version management.

I almost always develop code in PC test harnesses first. For complex code (file systems, UIs, etc) this is an obvious thing to do but it can also be done with all sorts of code (control code, GPS tracking algorithms, CAN drivers).... With suitable abstraction levels it is easy to make portable code that easily transfers between a PC test framework and the final hardware.

If the test harnesses have a GUI front-end that looks a bit like the final product then they can also be shipped out to marketing etc people as mock-ups. Let them play with the software and see if it really does what they want. Getting feedback early is way cheaper than getting it later.

The sooner you catch a problem the sooner you can identify it and fix it. The longer a bug remains live, the harder it is to identify and fix.

Where I can I like to put unit tests right in the main make file. Quite often short unit tests will run in a second or so adding an insignificant time to the build. If it fails the make stops. Pretty good way to determine that something you just did messed up!

Other tests take a bit longer and are impractical to put in a unit test and here is where Continuous Integration can work well. Continuous Integration takes the current code, builds it from scratch and runs a set of tests. Again the idea is to catch problems as soon as they happen.

The above strategies have saved me months.

While I have not tried it yet, the git source control system has a nifty binary search feature called git-bisect. Suppose you know that version A of code does not have a bug but version N does. How do you identify which version from B to N introduced the bug? git-bisect steps you through a binary search vastly reducing the number of versions you need to test. This can even be automated by writing a small test case for the bug then scripting git-bisect to **automatically** sequences through the binary search until the offending version is found.

4/9/2010 12:44 PM EDT

This is a great discussion. Changing some of the terminology just slightly leads right into OOP. Start with a few base classes, and define the function in terms of "tests" that must be run at every stage of design. As the design evolves add function in derived classes. The initial function is preserved by inheritance. Eventually hardware interaction is required. Hardware is a collection of functional blocks connected by Boolean networks. Classes defined by the hardware functions can be added, but the clocking has to also be added. Existing simulators run RTL and produce waveforms of hardware which is not suited for code debug. So far I have not gotten any traction with my CEngine, but it does have a prototype simulator for C statements if anyone is interested.

4/12/2010 5:36 PM EDT

Mr. Grenning:

I’m not sure what your reply has to do with my post, since I stated up front that I agreed with everything you said. Of course we both know that code rot happens one “Oh this little change won’t hurt” at a time, and I think that is what I said. Often new features have to be ready in time for a trade show, or an already announced product release so that the developer is figuratively working with a manager standing over him or her with a stop watch. In such cases a requirement would at least give the developer something to push back with.

As for “our professional duty” we get those lectures all the time, and some of us care and some of us don’t. My hope, as I said in my comment, is that a requirement for test driven development will force developers to think about keeping code clean and testable. We both know that if refactoring is put off until it is necessary it’s no longer possible with out destroying the schedule.

Edited by: ESD editorial staff: SRambo on Apr 13, 2010 9:51 AM

4/12/2010 10:01 PM EDT

1) Hopefully this is not used as an excuse not to document. Good documentation is the key to reusability and maintenance. In this hurry up and be quick approach I worry that the the desire to document after the work is done will create even worse documentation. The work is done there will be no desire to get the documentation right and the "test cases prove out the code so there is no need to". If this is work that is contracted out then you play to your contractors hand when it comes to a resourcing effort. Your contractor will be the only one who truly knows how the software works.

2) Loosely coupled software and embedded don't always go together. Embedded usually means getting the most bang for the buck which would seem to fight this concept. In addition todays systems are growing in complexity. In trying to take the burden off user and making user friendly forces more coupled / interactive modules.

Edited by: ESD editorial staff: SRambo on Apr 13, 2010 9:51 AM

4/13/2010 2:43 AM EDT

> 2) Loosely coupled software and embedded don't always go together. Embedded usually means getting the most bang for the buck which would seem to fight this concept. In addition todays systems are growing in complexity. In trying to take the burden off user and making user friendly forces more coupled / interactive modules.

Yet scientific research shows that tight coupling is one of the main causes for catastrophic failures in safety-critical embedded software. If you are making toys, hobby projects and commercial fluff applications where quality is less important than price, yeah then you can skip loose coupling. If you are making applications where catastrophic failures are bad, it is not so wise...

4/14/2010 9:58 PM EDT

Hi Larry

No worries, thanks for the apology.

Too bad TDD went bad for you. It does not sound like TDD's fault, but rather an integration problem. Integration problems are not unique to TDD. For my 30 years of development, late integration has always been a know problem. A misunderstanding of how the code should interacts with the real world is not a TDD problem. It's an integration problem.

I'd say you need both, tests and interface validation. As the interface is validated the tests should be corrected to match the lessons learned from integration. That way we get the benefit that TDD supports evolution mentioned by RubensLevy .

The fact that you are writing tests for code you are about to refactor puts you in an elite area of developers.

TDD tests do a few things for you:
- Help get it right the first time
- Help keep it right during refactoring
- Code rot radar early warning system
- Documentation that you can run to see if the code meets its spec.

thanks, James


Hi embedded_janitor

Testing does not generate the design. People do. Sorry if I was not more clear. As code looses its testability it is a warning of a rotting design. TDD helps designers see where decoupling is needed, where functions are getting too long, where nesting is getting too deep. If we listen to what the code is telling us as testability is eroding, we'll evolve the design and make it more modular and loosely coupled. code gets better overtime, rather than the slow degradation that is typical.

We're concerned with solving the same problems, with similar approaches. I hope your janitorial


Hi Mr Heretic,

Sorry to leave you with the impression I was disagreeing with or challenging you. I was not. My comments were a little tongue on cheek.


Hi KarlS

You might want to look into the SOLID Object Oriented Design Principles, popularized by Robert Martin. They give really good advice on keeping designs flexible. You might find that deep layers of inheritance hierarchies might cause some problems.

Hi R N

Documentation needs are really situation specific. I like to have a high level map of how the code is organized, standards followed, and technologies used. At the detailed level, TDD style tests can provide a very effective detailed design specification. Care has to be made to make the tests readable and people need to learn how to read them. This document can't lie, as comments and separate documents can. Running tests validates the document.

So, if I was not clear, documents what you need to, comment if the code can't speak for itself, let tests serve as detailed documentation.

Regarding tight coupling being a good idea. I'm with Ludin. I'm not for tight coupling, I'm for right coupling.


Thanks again for the discussion all. If you are interested in learning more about TDD for Embedded C, my book is in beta right now at www.pragprog.com

Will any of you be at ESC Silicon Valley? Come to my half day session on it and get your own first hand experience. https://www.cmpevents.com/ESCw10/a.asp?option=G&V=3&CPid=279

Let's also meet for a beer and a game of pool.

James