The Non-Quality Revolution

3/8/2010 2:24 PM EST

Well I think that now a days the price tag has become a deciding factor. Having said that, things such as quality, process, xyz-assurance, verification starts looking like a unnecessary overhead. 'Time-to-Market' is yet another culprit, pushing the idea of "Design today, Ship tomorrow".

3/8/2010 4:52 PM EST

This article reminded me of a comment Eli Goldratt (The Goal) wrote in one of his books (I believe it was The Haystack Syndrome): "Tell me how you're going to measure me and I'll tell you how I'm going to behave."

3/9/2010 7:05 AM EST

I think, quality in software is on the rise in safety critical systems. For long, that was only avionics but it is gaining ground in the general areas as well.
The price for it will only be paid due to the thread of costly law suits, where it should be paid for a better quality.

3/9/2010 9:26 PM EST

I happen to have first hand experience with the way firmware gets designed and verified in the Toyota world. In my case, simply firmware running in the entertainment components of the car (non-safety critical).

After being through days and weeks of joint DRBFM sessions it is quite hard for me to imagine that the current Toyota issue - if indeed a FW issue - is anything detectable by static checkers and thorough peer review.

By being in business with Toyota the code quality of my company was driven quite substantially. And it wasn't that bad to begin with.

I recommend to be very careful with assumptions about the code quality produced in Japan and especially when it comes to the Toyota ecosystem.

3/10/2010 3:02 AM EST

It is really not fair to throw eggs at the car manufacturers for doing nothing for software quality. Actually,
the automotive industry is leading the progress of embedded system quality. They are launching AUTOSAR as a big, international car standard to get rid of firmware problems, and they are behind MISRA-C, which seems to be growing into an industry standard for all embedded systems.

It is much more justified to ask why programming quality stood still during the whole 70s, 80s and 90s, when the computer industry was booming. All the big players in the commercial computer industry completely ignored software quality for 30 years, and are still ignoring it. Sure, they release fluff programs for project management, version handling etc, and they come up with fluff design tools and methods like UML. But they have failed to come up with an actually useful, general-purpose programming language. People are still writing in C and C++, horrible languages that are 30 years old.

You -can- write failsafe programs in those languages, but the amount of knowledge needed to do so is vast, you need at least 5 years of programmer field experience to even get close to it. Forget about fresh-out-of-school and/or hardware engineers dabbling in those languages on their own, you need to employ at least one software guru at each R&D department, with ridiculously detailed knowledge of C/C++. This is because the languages are so poorly defined, and they have countless tricks and pitfalls. They were great inventions back in the days, but 30 years later, we need something better.

3/10/2010 10:38 PM EST

I mostly agree with Lundin here. The language matters. The ongoing dominance of C/C++ and their relatives has been little short of a disaster for software engineering, and many of the tools are just an attempt to paper over the cracks. However I don't think that C and C++ were ever great. C was already primitive and confusing by the standards of the day even when it was first designed, if you can call the ad-hoc process that brought it about 'design' (and who can really be confident with C++?). Maybe it's understandable, given the constraints that its creators were operating under, but those are not appropriate constraints for the design of an embedded or a general-purpose programming language. Nor is it true that there have been no useful, general-purpose programming languages available. Ada was developed in the 1970s and first standardised in 1980, then as an ANSI standard in 1983. It is 30 years old, although admittedly widely available quality compilers lagged by a few years. It was standardised earlier than C was. Similarly, Ada-95 was standardised before C++. Even Modula-2, which has been available since 1980, is a reasonable option (not nearly as sophisticated as Ada, but much better than C). While Ada has (I think) become too complex recently (mostly due to OOP features), at least at its core was an attempt to be precise and consistent, and the Ada family (including the likes of Spark) is literally decades ahead of the C family in most respects that matter. Compare the extensive design and review process that led to Ada with the development of any of the C languages. But you see job advertisements today that say 'initially in Ada, but transitioning to C++'. It's insane.

3/11/2010 12:03 AM EST

In case anyone thinks that it is not feasible to use Ada to program a small micro...this is not true. Many of the differences between C and Ada are matters of syntax or of static semantics - examples include not silently omitting cases in a case statement, and allowing one to specify numeric types so that they can not accidentally be mixed up. These, and many others, are compile-time issues. Of course there are features that might not be practical on a given target, but Ada allows language restrictions to be applied, in a standardised way, in order to reduce run-time costs and increase the effectiveness of certain static checks. These restrictions take the form of 'configuration pragmas' that are specified in the language and written as part of the program text, not just switches on particular compilers and linkers. Likewise, many of the features from Ada's extensive set of standardised packages could add a fair bit of code, but only if you used them (like using library code in any program).

3/11/2010 3:21 AM EST

... It all depends on the domain you are working in ... In safety critical systems you have to prove that your code works. The car industry is for obvious reasons typically trying to produce well written and tested code, but the development process does not seem to scale as well as e.g. the Linux kernel, which has a phenomenal growth in lines of code without too many bugs introduced (forget what you heard about COCOMO with it).
But most embedded systems are not safety critical. What about those?
Unfortunately I see in my consulting and training activities more and more consumer electronics companies selling cheap, but not fully working stuff as opposed to more expensive, but working one. (I believe cheap and working don't exclude each other).
Are we in the meantime accepting bad quality firmware with the right price tag?

Regards,

Robert

--
Robert Berger
Embedded Software Specialist

Reliable Embedded Systems
Consulting Training Engineering
Tel.: (+30) 697 593 3428
Fax.:(+30) 210 684 7881
email: robert.berger@reliableembeddedsystems.com
URL: http://www.reliableembeddedsystems.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

3/11/2010 9:22 AM EST

I can not state if the phenomena is current, as I have not worked auto industry for about 5 years now. When I did work auto industry I noticed 3 distinct styles for requirements management. For lack of better nomenclature I will denote them the A, M and V styles.In each case I was working at a tier 1 supplier, to the focus of the observation is on the producer - consumer relationship. The mechanism by which sub-system characteristics are conveyed is the focus.

In style M, the tier 1, the OEM, followed an incremental-evolutionary development. Increments in the sense each interm developmental release was functionally incomplete. Evolutionary in the sense that the requrements for the completed system were changed from interm developmental release to developmental release. You really need to track such things with requirements database of some sort, but the tier 1 supplier clearly wasn't. This phenomena led to some interesting side effect. M's testing organization and M's development organization each often had different opinions as to the operational characeristics of the various deliveries (side effect of incremental-evolutionary, and a task indepencence firewall, ie each could contact the OEM without conveying discoveries to the other)

In style A, the tier 1, formerly owned by the OEM, the OEM provided interface specifications (which were quite well done) and performed qualification tests. Details such as exotic failure phenomena which needed to be addressed were not formally documented. Addressing such matters was handled in the style of an expert craftsman (ie we better cover this phenomena as it will be a problem). A rigerous FMEA was obvious done on the system (a safety device) and concievable the craftsman like behavior could have been little more than an interpretation of the FMEA. The essential issue here is that such observations are appropriately introduced as formal requirements rather than being handled by word of mouth.

In style V, the tier 1, formerly owned by an OEM, the OEM provided interface specifications (which were quite well done), functional behavior descriptions, and performed qualification tests. There was also an extensive UML model developed by V, which definitely described the architectue of the product line. There was unfortunately no tailoring document of the architecture to the specific product under development. Because the tier 1, V, was using a product line approach. There was a claimed operational baseline of code. I relatively quickly determined to be "buggy" the sub-system for which I was responsible. There was an excelent bug tracking system. It efectively transformed the operation ton into integrate (big bang) and fix bugs until no problems observed. The effort was limited by the rigerousness of the OEM testing.

You can't produce high quality software until you start tracking the requirements in a fashion independent of the people doing the actual work. You can't produce high quality software until the rigerousness of the testing can be determined.

The sad part of this entire observation is this. The concealed knowledge of A and the UML model of V are the bases for a pricing proposal and the tier 1 were (at that time) much more competative. Making that knowledge public would be paramount to revealing trade secrets. Something no smart businessman is going to do.

As for the contention - Quality is no longer job one - sillyness, it never was.

3/11/2010 12:04 PM EST

The three most important things for developing quality software are: requirements, requirements and requirements.
The software can only be as good as the experience and intuition of the person(s) writing the requirements. If there are no requirements for how the software should behave when things are less than perfect, the results could be problematic. Certified safety critical software can be tested to ensure that it meets all of the requirements and still be a fragile mess. With the right requirements, any language can produce quality software.

3/11/2010 12:40 PM EST

Sorry if I over simplify this but there is a point I want to make here.
Aside from all the concerns of the language or the process what about sound engineering and common sense. If I am driving a car with the cruise control on (aka - car, go this speed) and I touch the brake, the cruise control turns off. Good idea! In a computer controlled car, regardless of whether the computer thinks the user says to go fast (foot on accelerator) if they also touch the brake at least turn off the "go" juice and at best override the system and respond to the brake. Think very high priority interrupt.
I think all the issues mentioned above are very great issues that, YES, need to be addressed in the embedded realm, especially the management one. That said, sound engineering skills come before the tools and processes we use to design with. The best hammer in the world can be used to build some great things but it can also be used wrecklessly and destroy some great things.

3/11/2010 8:15 PM EST

Requirements do matter. Management, version control, review and testing matter. But implementation also matters. It is true that there are plenty of problems caused by inadequate requirements. But there are also plenty of problems caused by errors in implementation. Sure, it is possible to do a good implementation in a weak, confusing and error-prone language, and it is possible do a bad implementation in a stronger and more rigorous one. But that doesn't mean that language choice is not an issue. A better language will help to minimise a large class of errors (does anybody really deny this?). An exceptional carpenter could probably make a cabinet with a twisted square and a bent hammer, but any sensible carpenter would choose precise quality tools. Unless of course he is an embedded carpenter. Then he will probably just read books and articles subtitled 'tips and tricks for mastering the twisted square' and invest in gap detectors and putty.

3/12/2010 8:53 AM EST

willc2010 states "In case anyone thinks that it is not feasible to use Ada to program a small micro...this is not true.".

It is when there is no Ada compiler available for the chosen microcontroller, which is quite likely. Of course the reason there isn't one available is because there's no demand, because few engineers know Ada, because there are no Ada compilers available for small micros, because...catch 22.

Although there are tools to translate Ada into C which can then be compiled, I don't fancy debugging single-stepwise through translator-generated C code!

3/12/2010 11:37 AM EST

I know someone who works for a military contractor. He recently told me that when he checks modified software into the configuration management system, he then compares the file in the CMS with the file he checke in, becasue sometimes, the CMS enters the changes in ways that are not accurate. One more glitch in the chain of software development!!

In my own experience, getting requirements right, and then testing the final result thoroughly are two areas that often lead to difficulties.

3/13/2010 2:01 PM EST

Good article. The more our products and processes rely on embedded systems, the more this will come up. At some point, after more accidents and deaths, the deniers will come around to the need for proper software QA. As a new member of Professional Engineers of Ontario (PEO) Council, I expect to get into regulatory obligations for software (and other newer engineering specialties) under Canada's Engineering Act. It will not be easy since the profession is still focused on traditional civil, mechanical and electrical engineering practice.

3/15/2010 2:48 AM EDT

This article is about the contrast between what could and should be done, and what is being done. One thing that could (and I think should) be done is to get away from the dominance of the C/C++ language family. We do not need to wait for the development of yet another new language and tool set. We can make more use of what already exists. Ada, for example, exists now, is mature, is not that hard to learn, is available for many targets including some (but not all) micros, is well supported by modern tools, and is far better than C/C++ and their relatives. As the article says, the industry has to decide to do better than it does now. It would be little or no harder to target the Ada compilers at various micros than to target the C/C++ compilers at them. Overall software development costs with Ada are usually lower, not higher, and the results are generally better. The C family has so shaped the industry that a lot of knowledge of how to write software has given way to a grab bag of C hacks and tricks and debugging tools.

3/15/2010 3:48 AM EDT

Yeah we should get rid of that old Tyrannosaurus. A Brachiosaurus is clearly a much better dinosaur, because it has-...

Seriously, we need a completely new -modern- programming language.

3/15/2010 9:42 AM EDT

Quality == conformance to specifications. What nonsense. We all know that intuitively that is not the meaning of quality - we have been sold a bill of goods for decades now. BUT, if that is the way the game is now being played then I'm sure there was a specification for the Toyota gas pedal software that said "this software shall never cause the car to accelerate uncontrollably and kill people". How do you test that specification? Therein lies the problem with testing vs. specifications. You can be sure of one thing - during the development of this gas pedal software, someone, at some time, saw this behavior, or something similar - a rogue failure or unexplainable operation. But, obviously, no one found the cause of that problem and excised it from the system - you all know how this works - the new model has to be delivered to the dealers every October (a very hard deadline in the automotive world) so what do you do? None of the recent, thorough, testing has exposed this rare problem. Maybe it has gone away, maybe it got fixed somehow. You rationalize it away and the product gets shipped.

Another personal example. I currently work at a medical electronics company where a device malfunction can kill someone. The project is 3 years late!! Parts for the existing device have gone end-of-life, and the company was forced to do a respin of the old device to have a product to ship while waiting for this new device. The pressure on the software team to get done is enormous. About a year ago one of the software folks noticed an odd problem, but could not reproduce it. Obviously, there was no time in this schedule to persue this problem and no body was sure how serious of problem this might be since no one could take the time to properly research it. Do you think this could be our "gas pedal runs amuck problem"?

3/15/2010 10:48 AM EDT

Originally posted on Industrial Control Designline:

Good article. The more our products and processes rely on embedded systems, the more this will come up. At some point, after more accidents and deaths, the deniers will come around to the need for proper software QA. As a new member of Professional Engineers of Ontario (PEO) Council, I expect to get into regulatory obligations for software (and other newer engineering specialties) under Canada's Engineering Act. It will not be easy since the profession is still focused on traditional civil, mechanical and electrical engineering practice.

--Engineer62

3/15/2010 1:01 PM EDT

Interesting article written by David M. Cummings (JPL, NASA Pathfinder) in LA Times,March 11, 2010: "Haven't found that software glitch, Toyota? Keep trying:
An electronics problem isn't to blame for the sudden acceleration, say the carmaker's engineers. That's nearly impossible to conclusively determine through laboratory tests."

http://articles.latimes.com/2010/mar/11/opinion/la-oew-cummings12-2010mar12" target="_blank" style="">href="http://articles.latimes.com/2010/mar/11/opinion/la-oew-cummings12-2010mar12"target="new">http://articles.latimes.com/2010/mar/11/opinion/la-oew-cummings12-2010mar12

3/16/2010 12:00 AM EDT

Lundin writes:
"Yeah we should get rid of that old Tyrannosaurus. A Brachiosaurus is clearly a much better dinosaur, because it has-...
Seriously, we need a completely new -modern- programming language."

What is it about Ada-2005 or even Ada-95 that is not modern enough? It is unreasonable to claim that just because C and Ada both originally date from more or less the same era that they are therefore somehow equivalently 'dinosaurs'. This is a serious issue, though. Among the thousands of programming languages that have been developed there have only been a very small handful that have been the result of a serious, extensive design and review process, that have a comprehensive standard test suite, formal documentation requirements on the implementation, a system of standardised extension annexes (as opposed to informally grafted-on libraries and compiler-specific options), and so on. And how many of those were designed to support embedded programming? It takes years and real money to accomplish all of this. Of course Ada is not perfect. But it is a great deal better than the current situation, it is mature and proven, and it exists now.

Readers of this site know (and frequently complain) that endless changes without consolidation leads to endless problems. Why should that not also be true of something as complex and important as programming languages? Ada's ten-year update cycle seems to be about right - it tracks changes in programming technique without producing chaos. And the basically sound foundation makes it possible without too much upheaval.

3/18/2010 11:13 AM EDT

I would love to investigate Ada and Spark. Unfortunately, they are not available for the Coldfire processors that my company uses. We have size, power, and cost targets that drive our microprocessor selections.

As bad as C is, the number of compiler vendors available has forced a huge improvement in both the quality and the performance of the compilers. When there is only one vendor for the compiler, the quality is normally terrible. Our experience with the (sole source) compiler for the Analog devices DSP is so poor we are designing out the part. In the 1980s, the CMS compiler that the Navy mandated everyone use was so bug ridden that we would look at the assembler output of the compiler to find the errors and manually insert code.

The issue of software quality is multifaceted, and includes requirements, process, tools, and people. Any one of them can make software quality an illusion.

3/29/2010 12:04 PM EDT

1. We need a new/different 3GL language like we need a hole in the head. Quality becomes easier with an increase in abstraction level. We need to program at a higher level of abstraction. I currently try to use Executable UML (Mellor and Balcer) as much as possible. The main problem with raising the abstraction level is developer resistance.
2. More requirements are useless without better attention to requirements analysis. I often heard the reply, "We don't have any requirements.", to the question, "Where are the requirements for this code?", which should always invoke the response, "Then how did you know what to code?"

8/21/2010 11:27 PM EDT

You summed it up perfectly:

"Detroit scoffed at the idea for years till they learned that the up-front costs yielded massive back-end savings."

As a software engineer it is so hard to convince and prove to the company culture that this is a FACT and not fiction. The push for unrealistic delivery dates by the business also puts alot of pressure on us Software developers who are forced to cut proper QA processes. Yes a darn shame, programmers would really prefer a less stressful coding environment with proper QA staff, unit tests, and up front clear requirements.