Faster! Can we design embedded systems faster, cheaper, better?

6/2/2008 2:33 AM EDT

Hi Jack,
Great article, as usual.

I would like to add some comments:
1) Even if we would have the perfect processes and tools to predict how long it will take to produce our software, there is the famous "cone of uncertainty". It says, that at the beginning of our time estimation we are off by a factor between 4 and 0.25.
Only just before we deliver we have some clearer idea how long it will take to finish.
2) You know my opinion about reuse. In one of the projects I'm involved in, there is no other way. We are porting some middleware to set-top boxes with different APIs and operating systems and the only way to keep our code somehow maintainable is to reuse as much as possible. So only the machine dependent layer is (usually) changing per new set-top-box and for the application this is transparent.
3) Remember it's all up to the people. Tools (UML, static code checking,...) are usually shelfware, if you don't enforce them in your process and, more importantly, convince and train your people to use them.

Regards,
Robert

-------------------------------
Embedded Software Specialist
www.reliableembeddedsystems.com

6/2/2008 9:57 AM EDT

To me what is fasicnating is the demand on one hand for reusable software, similar to hardware building blocks, with the demand that software remain flexible.
Hardware is inflexible by nature, unless one gets into FPGA or CPLD, at least up to the I/O pins. One can change the internal design of an FPGA, as long as the IO remains where it was. Hardware blocks are defined by their data sheets as to functionality, and these typically specify behavior not just across parts, but different manufacturers. The amount of effort that goes into specifying a part in hardware would need to be attached to software, at which point much of the flexibility of software development is lost.
Our company has reused much of the software developed for older generation of product and carried it through to a third processor manufacturer. We probably match that 75-85% range mentioned in the article. At this point we are primarily working on changing the low level interfaces, but even so, every time we use a new compiler, we tend to find a few odd things in the code. We have used other tools (such as Lint) but as mentioned, each new compiler used looks at the code differently it seems, and locates some new warning or error.
I remember reading Sould of a new Machine when it was first published, and how well it reflected the operating environment of product development in my company. All you had to do was change some names and you had the same story going on. It hasn't changed in 29 years - a scary thought!

6/2/2008 10:00 AM EDT

All of your points about better tools are good ones, though we will never stop quibbling about what is "best" at any given time.

But, the tools discussion is beside the point. Tight, super-tight, and crazy schedules are a people problem, not a tools problem. It's tempting to think that if we learn to work (even) faster then the schedules will be easier to meet. All of human experience tells us this is not the case; if we work faster the schedules will get shorter to match. That's a different problem, less amenable to mitigation by engineering methods. Unfortunately.

6/2/2008 5:47 PM EDT

Great responses... As Paul Tiplady alluded to, it is all about the cost. The people who we work for don't understand (or necessarily care) about quality of the software. They barely understand a profit and loss sheet. They are ONLY interested in short term profits. We are never going to be able to educate them, and it is getting worse (as they are getting more shortsighted).
We have known about tools and techniques to improve the development of software for the last 30 years, but they rarely get used, they are a failure!
Because no matter how good a tool is it is worthless if it isn't used.

We work to the level of the hardware, it will always be custom so as to be as cheap as possible (it's what the accountants care about), so the software will be custom as well, except for the highest, most abstract routines in a system (certainly less than 20%).

We have to stop discussing failed approaches, these tools, no matter of good they look on paper, are never going to be used industry wide. Wipe your eyes, crying time is over, there are no shortcuts. Now get back to work, that deadline is almost here!

6/2/2008 6:57 PM EDT

Well... all of the above, some of the above, and none of the above. In general, all those fancy computer aided tools will get you is as good as what is put in. You know that old GIGO buffer effect!

I can tell you this though, the days of the lone embedded engineer, working on that one project, over and over ... and over, are starting to come to an end. The newer methods are starting to prevail. And by this, I mean team-based firmware development. Unfortunately, this style tends to work best if sufficient resources are relegated up front to create some kind of framework or API that engineers of the future may easily leverage into new products with new features in a shorter amount of time.

But alas! Usually marketing becomes too impatient and starts hamstringing the process by making promises that cannot be kept to the customer!

6/2/2008 11:14 PM EDT

There are some interesting themes running through this discussion, with cost, project management, tools and reuse. A few comments: First, as a high-tech industry, we should be ashamed of ourselves if the best we can do to solve problems is to throw more people at the problem. We should be using technology to solve problems. Second, some of these problems have been faced on the hardware side, in the development of the integrated circuits that our embedded software runs on. Reuse is more prevalent in hw design, because over the last 10-15 years standards for reuse have been developed. Part of those standards is verifying the reusable module, and this is one place where tools and methodology can make a difference, a big difference. Third, project management is shooting in the dark because there are few metrics developed to measure progress, and therefore be able to manage the development process. Things we need to do.

6/3/2008 3:53 AM EDT

Paul Tiplady hit the nail with "how cheap can we make /this product/ and still run a profit" - because, making it any cheaper usually provokes a myriad of quality problems - which just shift the cost to a later date.

Second point - my own experience - is that many embedded software projects originate in companies with a strong mechanical or electronics background. Manufacturing mechanical or electronics assemblies is way different from designing software - this still seems a secret for management. Managing the production flow of real parts is far more easy than the production of imaginary goods, aka software. While processes and methods can be applied you can not force creativity in coding when the available solution space is large. Hence the recent success of languages which limit the available solution space, like MISRA-C and more far reaching, yet labled exotic constructions to prevent buggy coding in the first place.

My gut feeling is that software generation from domain specific models will be the road to follow.

6/3/2008 8:18 AM EDT

I couldn't agree more with B Kockoth on the second part of his speech. I've always said that re-use in software doesn't reach a similar level to what is reached in electronics because it has too much flexibility on the *interfaces*. That's why re-use is very difficult, and that only changes when you reduce the flexibility at the borders (interfaces). On electronics you don't have that much other than Volts & Amps and just a few well-known ways to make adaptations of those, while in software you can use many different ways just to represent an integer value, all of them incompatible from a high level point of view!

That's why languages like TCL are so good at fast integration of different components: it supports only 1 base data-type, which drastically reduces the number of formats you can pass data through interfaces.

6/4/2008 11:23 AM EDT

A good article but B Kockoth's 2nd point is very pertinent. It is still so often the case that organisations demand design efforts during H/W development and then simply 'expect' software to wave a magic wand.

Tools & methodologies only fail when you expect them to be panaceas to all of your problems or that any one tool or method is a perfect & absolute fit for your business case.

We must adapt and 'cherry pick' the things that best suit us, our workforce and our environment and then ensure consistent application of our chosen process(es).

Of course, we have to do this while contnually staring down the accountant's gun - but somethings will never change.

6/4/2008 11:35 AM EDT

I love the part about the crazy deadlines that are set by marketing. Normally the people that set the deadlines do not do the work.
I agree with the comment above. From my experience as well many embedded systems are developed by companies that did/do manufacturing or electronics. In the case of the company i worked of it was manufacturing. Which is nothing like software development. Most of the management set the deadlines based on building the parts and the software needed to fallow that deadline? The results are bugs and problems. These companies need to take tips from software developers like google as to how to plan projects.
At the same time engineers need to move to using C++ and Java. They make it much easier to re use code and you can code faster? The only people that do not want to move off of C seem to be the older guys who say that C is faster and better to use?

6/5/2008 4:53 AM EDT

1. For the most parallel programming model you can think about the Informational Individual, as the final frontier in the software thinking!! For beginning http://gheorghematei.blogspot.com

2. "A class is a new data type." OK! But in the future we will solve problems without "new data types"!!!!!!!!!!!!!! I'm working to a new programming paradigm in software and ... without classes!!!!

3. The multi-core is not the solution. We are thinking as in the current model of programming. Here is the mistake. For the future we will work on a new software model -- the Informational Individual! The processor will build on a structure and functionality of Informational Individual not a C++ language. Now we are thinking as in 1970. Now the problems are in the kind of thinking in software. Here is the barrier.

6/9/2008 1:27 AM EDT

During my time in the consumer electronics world, schedules really came in two flavors-- ones ending with new products ready for sale in time for "Dad's and Grads" (May-June) and ones ending with new products ready for sale in time for "The Holidays" (Nov-Dec).

Into this mix you then threw "CES" (possibly "CeBIT" for good measure a couple months later), which invariably added a new milestone... Some sort of one-off demo device for the tradeshow booth that looked and worked just like the one that wouldn't be ready for production for another three months. No problemo. You'd be amazed what I can build out of hot-glue 24 hours before my flight leaves.

Those lovely Gannt charts in MS Project (and the like) then became an exercise in "pulling in the schedule". (Or as I liked to call it: "being late sooner".)

The duration for a task tended to be set more by the time available for it in the schedule, rather than the duration of the work required to complete it. It's pretty easy to see where *that* leads. (Lots of "death march" at the end!)

I *am* a big advocate of using a tool like Microsoft Project (or name your poison) if only because the actual exercise of creating a detailed schedule really does force you to think through all the dependencies and tasks that are really required. ...and it makes the Project/Program Managers feel loved. ;-)

-Clay

6/9/2008 10:59 AM EDT

For me it is clear that we can only improve by levering our understanding for complexity by using abstraction i.e. Modeling combined with Automation (i.e. executable mdoles and code generation right to the target). Thus investment in Tools and Methodology. I have been introducing this into the market for over 8 years now and results are staggering. Many projects would have not come out in the time given and the quality required using traditional methods. The types of devices have been various, from sowing machines, wireless mobile devices, machine control, motor control, medical devices ...You need to however to invest in the right tools such as Rhapsody in order to have the gain ... however the tool will not do the job for you, you need to apply it correctly ... and then you will ease your work by factors ...

6/9/2008 12:36 PM EDT

Jack, good to see you referencing Software Engineering greats, Brooks and Boehm.

Professional engineers tend toward "BETTER, Faster, Cheaper"; like a golf-swing, the technique has to be perfected before power and distance are achieved.

6/10/2008 2:51 PM EDT

"Any idiot can write code"? Indeed. It seems several of them do, and I'm amazed that Jack,

whom I've come to expect better from, would say such a thing even in jest. It's amusing that

thirty years into the search for the holy grail of code reuse, many among us still refuse to

consider that maybe -- just maybe -- that beast doesn't exist, or at least doesn't deserve

some of its more fabled reputation. Is that heresy? Oh, it would be nice if we ever reach the

point where application design amounts to stringing together a new permutation of trusted

library modules. But wanting it doesn't make it so. What if the harsh reality is that

excellent code really is more like a great piece of music after all, and less like the

instrument on which it's played? What if just any old idiot can't, in fact, write it? Is that

maybe just possible? Lest anyone think I'm against programming at higher levels of

abstraction, I'll tell you I'm not. I think it's essential. But I'm under no delusion that

incremental gains in this area are ever going to placate those who insist you can invent on a

schedule, or that the solution to insane schedules is to ratchet up the technology. All too

often, doing this just obscures underlying problems in the code base, and how long is it

before we have such a high technical burden that nobody really understands any of it? No, as

helpful as reuse may be, it's not a panacea. Engineers are a truthful lot, in the main, and

the unpleasant truth is still: You want a baby? Fine. It's going to take nine months.

V. Owens
Hardware Engineer

6/11/2008 5:35 AM EDT

A baby in 9 months? But if we put two mothers on the job, and allow a bit for communication, we must be able to get it through in 6 months...

6/11/2008 10:40 AM EDT

E-mail from a reader:


Dear Mr. Nass:

In his June column, Jack Ganssle ("Faster!", June 2008) asks whether the software IC is even possible. It is easy to demonstrate that it is not. All software components (particularly the off-the-shelf kind) involve tradeoffs. If you use small components, you will need a lot of them, and hence a lot of glue logic--thus defeating the purpose. If components are large, they are either rigid and unlikely to fit your needs, or they are highly configurable and therefore very complex to understand, integrate and maintain. Then there are the issues of source code availability, royalties, and the fact that you are more dependent on outside vendors.

But the software/IC analogy is actually flawed at a deeper level: If software were like integrated circuits, the best-designed systems that I have worked on would have their equivalent in several hundred ASICS, each approximately the complexity of the Z80, but all different and highly interconnected. Furthermore, during development (and after release), both the ICs, the interconnections among them, and even the number of pins would be changing on a weekly basis as the market's needs evolve. Hardware ICs simply are not a good model for software.

Additionally, if a company wants to differentiate itself from the competition, it has to produce most of its software internally, so that the effort it expends and the expertise it possesses will show up in the end product. If a software system were 95% off-the-shelf components and 5% glue logic, it would be easy to recreate, and would demonstrate that the company had nothing to offer itself. Internally-written components are a good idea, but expensive to get right. Unless you are using the same hardware, user interface, etc., over and over, you will find yourself writing large numbers of new components for each new system, hence nothing is gained.

Software development is the encoding of messy, domain-specific, real-world knowledge into machine-usable form, and is unique among human endeavors. Its incredible flexibility is also unique--and is its greatest strength. Our desire to formulate simple solutions just proves that we don't really understand it yet. The sooner we accept that it is different, the sooner we can move forwards.

Grant D. Schultz
Senior Software Engineer
Mission, KS

6/11/2008 11:07 AM EDT

One of the strengths of Fortran (and C for that matter) is a collection of totally debugged subroutines. This code reuse is so casual to be not noticed and yet has been wonderful for productivity, such that we casually consider it to be part of normal productivity.

What would be great would be an expansion of this to the usual and maybe unusual applications, an code library of wondrous size.

Sean

8/4/2008 3:09 PM EDT

The following comment was submitted to the editor in chief by e-mail and originally appeared in the Parity Bit section of the August print issue of Embedded Systems Design magazine.

I read with interest Jack Ganssle's "Faster!" and the feedback it generated. (Breakpoints, June 2008, p.53; Parity Bit, July 2008, p.9, all available at www.embedded.com/208400913)" target="_blank" style="">href="http://www.embedded.com/208400913"target="new">www.embedded.com/208400913) For most of my 30-year software-development career, I've heard the constant message that there aren't enough programmers, software is too expensive, we need reuse, more abstraction; and seen various attempts at making it happen. When I started, programs had hundreds or thousands of lines of code and memory was in KB. Now programs contain millions of lines of code and occupy GB of memory. Little else has changed. Doomsday never arrived, nor did a solution.

First, neither the 65-hour workweek nor the "mythical man month" are programming or computer issues. Both are management issues and can be solved and have been solved in organizations willing to address them. I really don’t need a new cell phone every year or two with yet more features I can't use unless I spend two days reading the manual. The time crunch for most of this stuff exists only to benefit the market position of the company peddling the product, which in theory is supposed to benefit the prosperity of society, but in practice only has a positive impact on those well above median income.

Regarding Jim Ford's question, "Why is there so much difference between hardware and software?" Hardware got off to a fundamentally different start, thanks to packaging. Early devices had to be task specific because of the limitations of what could be put on a piece of silicon. That forced systems to be made from a collection of components, which in turn had to have a standardized interface. Even today you can buy just about any 8- or 16-bit CPU and some SRAM and with the addition of 8 or 16 data lines, a dozen or two address lines, and three control lines have them communicate. The physical package boundary reinforced this.

Of course, if you were to build a 3-GHz Pentium desktop out of such components, it wouldn’t run anywhere close to 3 GHz and would consume thousands of watts of power. The inefficiencies of converting every signal to a standard interface on that scale, with enough drive to reach all destinations quickly enough would consume far more power than any useful work would. We consider such totally unacceptable, and hence--especially in the embedded world--ICs have increased in density and integration, eliminating this inefficiency, and FPGAs, ASIC, and IP have become increasingly a part of the landscape. Even with IP, though, there are standardized interfaces borrowed from our history, just simpler, less power-greedy ones. They still add to the size of the die and its power consumption, and the designer must decide to what extent to work at a lower level and eliminate some of them in the interest of size and power budgets or leave them alone in the interest of schedule deadlines and costs.

Software has taken exactly the opposite approach. It started without interface standards. Software reuse has suggested the need for such standards, but like hardware, there is a price to be paid for that standardization. Likewise higher-level languages, while offering more abstraction, also extract a price in efficiency. With clock speeds in the GHz and memory in the MB and GB region, the tendency is to move in that direction with abandon. I was interviewing some consultants recently for a project I needed to outsource. One recommended doing it in Java. I asked him about efficiency, and he assured me it was so close to C and C++ as to be down in the noise and sent me some links to studies giving information. While I had to admit the numbers looked better than I expected, the general consensus of the papers was that the performance hit was no more than 2:1. Well, on an otherwise idle, state-of-the-art desktop computer that may not seem significant, but as I read the concerns about the large percentage of our electrical power being consumed by major server farms (which happen to be very tempting candidates for this abstraction), the decision to use Java instead of C could make a 2:1 impact on their electric bill! That is not down in the noise!

I’m all in favor of leveraging the productivity of programmers (including my own), but we need to come to terms with some hard facts. The first is that interpreted code (including JIT compile) by definition can never be competitively efficient for production. Anything that can be resolved once at compile time should not be reinvented every time a function is called. This is a matter of fundamental physics and cannot be circumvented. Why are we throwing so much effort down a guaranteed rat hole? Second, that energy should be more productively channeled at techniques that increase the level of abstraction while providing optimization that closes the performance gap. Higher levels of abstraction offer opportunities for optimizations that the programmer might not think of or recognize and that would be precluded by details that must be specified at lower levels of coding. Maybe a state machine with an unusual assortment of state values would work more efficiently than anything the programmer envisioned. Maybe a lookup table would be a better solution than a numerical algorithm or set of branches. Its similar to what a switch() statement offers a C compiler--the freedom to consider different ways of accomplishing the task. Sadly, we’ve probably spent 100x as many man-hours tweaking JIT runtimes as we have doing this sort of exploration.

As far as code reuse, the only technique I know of that works is to write code and reuse it. I've been doing it for 25 years now. The products I currently ship contain pieces of code I first wrote that long ago. To be sure, I give some thought to the interface when I write. Yes, some of the pieces have evolved over the years. Yes, there is some time spent on adapting it to the new need, which detracts from the productivity gain. However, there are significant gains in two areas: first, the software architecture of the project is already specified because of the reuse so I don't have to invent it from scratch every time, and second, every chunk of code dropped in from a previous project is a chunk of code that has been debugged and field tested, hence the debugging cycle is shortened and the reliability increased.

I have standardized on a single CPU family so even my driver development is minimized, as is my learning curve, both on the hardware and the tools. Every time a new project comes in the door, I look for a recently completed project most like the new one and copy the entire project into a new directory. Then I delete what doesn’t apply and start changing and adding to what is left. I can turn out a fully debugged control project with graphic LCD and touch, serial, and Ethernet in a matter of a few weeks. I can also port to new hardware in a matter of days--all in C with judicious use of assembly.

That’s my version of reuse. It’s the only version I know of that really works!

--Wilton Helm

Embedded System Resources

Golden, CO

null