Design Article
Getting disciplined about embedded software development: Part 2 - The Seven Step Plan
Jack Ganssle
3/9/2010 12:47 AM EST
That tool is: an absolute commitment to make some small but basic changes to the way you develop code .
Given the will to change, here's what you should do today :
1. Buy and use a Version Control System.
2. Institute a Firmware Standards Manual. .
3. Start a program of Code Inspections. .
4. Create a quiet environment conducive to thinking.
More on each of these later on. Any attempt to institute just one or two of these four ingredients will fail. All couple syneristically to transform crappy code to something you'll be proud of. Once you're up to speed on steps 1"4, add the following:
5. Measure your bug rates. .
6. Measure code production rates. .
7. Constantly study software engineering.
Does this prescription sound too difficult? I've worked with companies that have implemented steps 1 to 4 in a single day. Of course they tuned the process over a course of months. That, though, is the very meaning of the word "process" something that constantly evolves over time. But the benefits accrue as soon as you start the process. Let's look at each step in a bit more detail.
Step 1: Buy and Use a VCS.
Even a one-person shop needs a formal VCS (Version Control System). It is truly magical to be able to rebuild any version of a set of firmware, even one many years old. The VCS provides a sure way to answer those questions that pepper every bug discussion, like "when did this bug pop up?"
The VCS is a database hosted on a server. It's the repository of all of the company 's code, make files, and the other bits and pieces that make up a project. There's no reason not to include hardware files as well—schematics, artwork, and the like.
A VCS insulates your code from the developers. It keeps people from fiddling with the source; it gives you a way to track each and every change. It controls the number of people working on modules and provides mechanisms to create a single correct module from one that has been (in error) simultaneously modified by two or more people.
Sure, you can sneak around the VCS, but like cheating on your taxes there's eventually a day of reckoning. Maybe you'll get a few minutes of time savings up front inevitably followed by hours or days of extra time paying for the shortcut.
Never bypass the VCS. Check modules in and out as needed. Don't hoard checked-out modules "in case you need them." Use the system as intended, daily, so there's no VCS clean up needed at the project's end.
The VCS is also a key part of the file backup plan. In my experience it's foolish to rely on the good intentions of people to back-up religiously. Some are passionately devoted; others are concerned but inconsistent. All too often the data is worth more than all of the equipment in a building, even more than the building itself. Sloppy backups spell eventual disaster.
I admit to being anal-retentive about backups. A fire that destroys all of the equipment would be an incredible headache, but a guaranteed business-buster is the one that smokes the data. Yet, preaching about data duplication and implementing draconian rules is singularly ineffective.
A VCS saves all project files on a single server, in the VCS database. Develop a backup plan that saves the VCS files each and every night. With the VCS there's but one machine whose data is life and death for the company, so the backup problem is localized and tractable. Automate the process as much as possible.
Checkpoint your tools. An often overlooked characteristic of embedded systems is their astonishing lifetime. It's not unusual to ship a product for a decade or more. This implies that you've got to be prepared to support old versions of every product.
As time goes on, though, the tool vendors obsolete their compilers, linkers, debuggers, and the like. When you suddenly have to change a product originally built with version 2.0 of the compiler—and now only version 5.3 is available—what are you going to do?
The new version brings new risks and dangers. At the very least it will inflict your product with a host of unknowns. Are there new bugs? A new code generator means the real-time performance of the product will surely differ. Perhaps the compiled code is bigger, and no longer fits in ROM.
It's better to simply use the original compiler and linker throughout the product's entire lifecycle, so preserve the tools . At the end of a project check all of the tools into the VCS. It's cheap insurance.
When I suggested this to a group of engineers at a disk drive company, they cheered! Now that big drives cost virtually nothing there's no reason not to go heavy on the mass storage and save everything.
A lot of vendors provide VCS. But today the most popular is the open source Subversion. Another open source product, Trac, gives Subversion a Wiki front end with a better user interface.
|
|
|
|
|
goldenpiggy
3/11/2010 4:58 AM EST
Jack, thank you for a great writeup -- wouldn't it be great if this can be made into posters and hung in cafeteria walls of every software shop? (in similar fashion to equal opportunity employment posters.)
I may be going off topic here. Regarding the "checkpoint your tools" section: there are probably old tools and compilers that refuse to run on modern operating systems, or may even be "node locked" to some particular dedicated build machine that's well, somewhere in the building. (Unix-based tools are probably less affected than PC-based tools.)
Perhaps preserving the tools is not enough, and one must also preserve the build environment (as in the machine and OS that the tool runs on)? But in many organizations, things just get thrown out. I think this is one area that virtual machines (e.g. VMWare, which I have no affiliation with) can help preserve an old build environment.
Suppose you have a compiler toolchain that only runs on Windows NT or relies on some DLLs not present on more modern OS. One can create a virtual machine of that NT machine, complete with the compiler tools already installed, and have it for posterity. In the future if someone needs to make changes, there's no need to worry about locating that NT build machine (how many NT machines can you find in major companies these days?), finding the install CD, etc. Just run the virtual machine on your latest and greatest PC. The build time will probably be faster under in a virtual machine given today's faster processors.
Virtual machines are popular in the IT camp, but could prove useful in embedded development.
All the best.
Sign in to Reply
tfc
3/12/2010 10:44 AM EST
Great article. What I find interesting is the Peoplware statement that 6 month engineers are as productive as seasoned engineers (Probably the only line management locked on to. I have seen management use this same argument when talking to old timers when I was new). Technically this may be the case, but in practice, the seasoned engineer wins because there is more to engineering than skill. From what I gatherer from the article, engineers must know how to expertly handle unique expectations that other people have created. This is where skill ends and experience takes over. I especially find the idea that you must covertly do what is right rather sad. If you succeed, it is managements success, if you fail, it is your fault and are unemployed (for years). Several quotes like happiness on the job depends on how sane the boss is, "Old age and treachery beats youth and skill every time, and Bad planning on your part does not constitute an emergency on my part come to mind.
Sign in to Reply
K1200LT Rider
3/15/2010 11:28 AM EDT
Ever since trying (and being impressed by) Sun's VirtualBox, I got the impression that virtual machines may really be the way to go when it comes to preserving the development environment. However, I think it will also bring up new licensing questions. New rules will have to be made about using virtual machines since a single one could contain a copy of a licensed OS as well as who-knows-how-many other licensed app's. The difficulty with getting this worked out might be illustrated with another experience I had with my present employer (a large company). I was able to have the appropriate persons create an archive area for 3rd party software on a big document control system. Even though access to any files could be controlled, they were only willing to allow storage files that did not require any kind of licensing. That severely crippled my original idea of keeping copies of ALL 3rd party software online. Right now, any original, physical media are kept in a CM file drawer, but anyone with a "need" can sign them out. So, what's the difference between signing out a disc or getting permission to go to a particular "3rd Party" folder and downloading it after getting the data gatekeeper-person to set permissions for you? Processes haven't caught up with how things could or should be done today.
I also asked our National Instruments sales rep about finding out how the licensing rules would apply to storing one copy of LabVIEW within multiple virtual machines. That was 2 months ago, and I haven't heard back.
Does anyone know of a company successfully using virtual machines to preserve development environments? I truly think this is the way to go.
Sign in to Reply
WflynnJr
3/18/2010 8:37 AM EDT
Jack,
Good stuff! Peopleware is right on, but I think you might want to go back and re-read it. Don't they say something about listening to music while working, and how it steals all your creative brain cycles? It has been my experience that this is very true. In fact I had one young engineer that I thought no amount of coaching, training, or developing was going to save. He continued to produce low quality code with little creativity or innovation. Then I insisted we try an experiment - he was to work one month without listening to music while working. The results were nothing short of dramatic.
So, listen to music to drown out the surroundings? Probably no. Pink noise? Waterfalls? Rainstorms? Ocean surf? OK.
Thoughts?
Sign in to Reply
careerEE
3/18/2010 11:21 AM EDT
Jack writes on VCS: "There's no reason not to include hardware files as wellschematics, artwork, and the like."
I agree, use, and endorse VCS for hardware design. But be aware of a big use-case difference between software and hardware: Software sources are text, and objects are binaries. For all hardware CAD tools that I am aware of, the sources are binaries (schematics/layouts) and the objects are text (gerber/drill files). Implicitly, the branch-and-merge paradigm does not apply, since we cannot diff binary sources. As a result, we literally track changes here, by typing their descriptions into spreadsheets. Yuck. I welcome any suggestions on this problem.
Sign in to Reply
WflynnJr
3/18/2010 9:03 PM EDT
Most CAD programs can export their binaries to some type of readable ASCII format. By using these you may be able to get your branch and merge concept, and you may get another plus: version independance and/or portability. Our CAD system can import the ASCII versions of PCAD and ORCAD among others, but it CANNOT import the binaries. So, if you ever switch CAD programs, your data may be more secure as it may transport.
Sign in to Reply
K1200LT Rider
5/6/2010 9:14 AM EDT
>> Don't they say something about listening to music while working, and how it steals all your creative brain cycles?
I have never seen this mentioned anywhere, but I actually wondered about this to myself because I am constantly having to put headphones on to drown out distractions around me. Maybe I'll have to try the surf or rain noise to see if it helps my concentration.
Sign in to Reply