Successfully Using Linux and Open Software in an Embedded System Design

Fueled by time-to-market pressures and the need to reduce overall system cost, many developers are migrating to Linux and open-systems software in their embedded systems designs. This open-systems approach allows developers to leverage freely available software, and spend their resources on their specific area of expertise to add value to their products. Embedded systems present unique design challenges such as: memory restrictions, booting from flash, diskless operations, and requirements for deterministic operation. There are some real challenges that face an embedded designer who wishes to use Linux in his product. This article presents questions developers should ask while planning embedded designs, how Linux stacks up, and how to overcome some of those challenges.

Is the operating system available for the desired microprocessor?

There are several criteria used to evaluate and choose an operating system for an embedded system or product: availability, resource usage, software available, feature set, reliability, and performance. Simply whether the operating system is available for the chosen embedded microprocessor can be the major decision factor. Because of the short time-to-market of many of today's embedded products, a port of an operating system to a new CPU or processor board may take too long. Having an operating system that already runs on the board saves the time and the expense of porting it.

The most popular version of Linux by far is the x86 version for PC-compatible computers. PCs are inexpensive and can serve as both a development host and a target. Most of the Linux developers around the world have PCs and develop for it first. Many embedded systems use x86 processors. There are many variants x86 from very fast to very low power. A complete PC compatible computer can fit on a single chip. For some embedded developers an x86 version of Linux is all that is needed. But in some cases an x86 processor is not the best choice. In the ultra low power range other processors such as the ARM or Hitachi SH processors can provide the same performance level and use less power. For some applications micro-controllers with PowerPC CPU cores, for instance, have the right built in I/O and coprocessors to reduce hardware costs. For some performance ranges there are MIPS processors that have a better prices performance ratio than any x86 processor. So the embedded microprocessor market is still fragmented. In fact it is as fragmented as it ever was and there is no sign of consolidation.

The good news is that Linux has been ported to many different embedded microprocessors including the PowerPC, MIPS, ARM, StrongArm, and SH. The bad news is that these ports of Linux do not yet enjoy the amount of third-party software that the x86 port of Linux has. Now that there are vendors for embedded Linux making periodic releases of their Linux distributions for non-x86 platforms, the situation should improve. A growing number of third-party software vendors have pledged support for Linux on a variety of embedded processors. But customers provide the strongest motivation. Inquiries about the availability of an application, or protocol stack, for Linux on a specific processor may speed the porting processes.

Can the operating system support headless and diskless operation?

Embedded systems rarely have a computer operator and most run as headless systems, meaning that there is no operator interaction required to start and run the system.

Linux was designed originally as a desktop operating system and it follows the UNIX model. A text console is used to interface with the computer operator. The console is used for controlling booting the operating system, giving the system commands, and monitoring the systems activity. Embedded systems rarely have a computer operator. So an embedded Linux must be setup not to rely on one. Linux like all flavors of UNIX, uses the file system to store and locate executable programs, and persistent data. Even system devices are referenced by special files on in the file system. A file system on some type of disk device is an integral part of the Linux execution environment.

Making a headless Linux system is not too hard. The console device can simply be a null device. There is no operator interaction required if the startup scripts simply start the application programs instead of login processes. Monitoring can be done remotely if needed by setting a pseudo TTY as the console once a remote login is established. Linux does support graphical user interfaces with X-windows. But X is strictly optional and not required for the operating system to run.

The problem of running without a disk can be solved in a few ways. There exist Flash memory devices that emulate IDE disk drives. These devices can be physically very small, not much bigger than the IDE connector itself. Linux also includes a RAM disk driver. This is a device driver that uses some amount of available system memory to emulate a disk drive. A RAM disk can be used for data files that do not have to be persistent across a system reboot. It is also possible to use the RAM disk driver to access a file system in ROM. If the embedded system has Flash memory that does not emulate an IDE drive, a flash file system can be used to treat the flash memory like a disk. A good flash file system will use wear leveling to keep flash block erases to a minimum.

Can the OS Operate With Limited Memory?

All operating systems require some resource usage in order to operate. The main resource required is computer memory. Some operating systems have a larger memory "footprint" than others. In embedded systems where both RAM and ROM is a precious resource, an operating system with too large a footprint must be ruled out.

By default Linux has a big memory footprint, too big for most embedded systems. The Linux distribution is hundreds of megabytes. A typical Linux kernel is 1.5 megabytes uncompressed. RAM use by the kernel for a default configuration is over 4 megabytes. Linux also requires a file system for operation, and if the embedded system does not have a disk this means more memory will be used in the form of a RAM disk.

The solution to the memory footprint issue of Linux is through careful configuration of the Linux kernel. The Linux kernel is modular, and to a certain degree unused kernel facilities can be configured out. Also, kernel resources can be configured such as the maximum number of processes or open files on the system. By configuring the operating system modules and resource, the size of the ROM and RAM footprint can be reduced. It is possible to configure an "off-the-shelf" x86 Linux kernel to be 259K uncompressed. A minimal ROM disk to provide a file system for this kernel can be as small as 102K. The total RAM usage can be under 4 Mbytes. A Linux capable of TCP/IP networking is larger. The kernel with networking configured is about 370K, and the required ROM disk is 740K (both uncompressed). RAM usage in this configuration of Linux is under 8 Mbytes. This is not as small as an embedded RTOS can achieve but small enough for many embedded devices.

Are there software development tools for embedding the operating system? Is the operating system compatible with other application software needed?

When developing embedded applications, developers must use software tools such as compilers and debuggers and these tools cannot commonly run on the embedded target due to resource constraints. Embedded development requires running the software development tools such as a compiler, assembler, and linker on a host computer then downloading them to a target computer for execution. The host and target may have different CPU types. In such case a compiler and assembler intended for native use cannot be used as a cross compiler and assembler. A cross compiler will have to be built or acquired that can run on the host and create machine code for the target.

Embedded operating systems, like desktop and server operating systems, are many times chosen not for their feature set but for the software available to run on them. Many people run the Windows operating system on their desktop computer because they can get the word processors or spreadsheet programs they need to use for Windows. Embedded systems designers need software beyond the operating system. They require protocol stacks, middleware, device drivers for special hardware, and sometime the actual application software. Whether or not this software is available for the embedded operating system they are considering without the need of porting is a very strong factor in their operating system choice. The feature set of an operating system becomes an important factor if the application software is going to be developed for the embedded system and is not already available. The availability of a certain feature may reduce the software development time. Of course, software that needs to be ported has been written assuming some set of features of the underlying operating system. If those features are not available, it may mean a long and costly port.

Most Linux software development is done natively. Programs are written and executed as well as debugged all on the same Linux system. Once the program is working its executable image or a "package" containing its executable image can be copied to other Linux computers for execution. This method of software development can be used for embedded systems as well providing the embedded system has sufficient memory and disk space resources to run the development tools. This typically is not the case, however. So cross development is required.

For cross development, the popular GNU compilers can be used. If the host and target CPU byte order is different such as an x86 host and a PowerPC target, sometimes bugs will surface. When using cross development, some method of downloading a kernel image to the target must be chosen. If the target can access a disk drive (and can boot from disk), the drive can be mounted on the host computer to write the kernel and file system and then moved to the target for booting. If the target can boot an operating system over a network interface, the host computer can simply run a boot server and a file server and the target can do a network boot. However, if the target can only boot from ROM or Flash memory, a special utility may be required to create a single load image given a Linux kernel and a file tree.

The easiest way to get a workable cross development environment for Linux is to pick up one of the Linux distributions designed for embedded use. These distributions have pre-built cross tools for several embedded processor types. Any byte order related bugs have usually been fixed. These distributions typically include booting code for the target as well as tools to configure and build Linux runtime images that can boot over the network or from Flash.

Cross development also requires cross debug support. In a cross debug environment the debugger runs on the development host. The debugger included on Linux distributions is GDB, the GNU debugger. GDB can run native or cross. In native mode GDB executes or attaches to the executing application to be debugged. GDB then can examine the variables and data structures of the application program or library through the /proc interface. GDB can also trace the program, be notified of signals being sent to the executing program, and send signals itself. GDB reads the symbol information from the executable file of the program and can also display the source from the programs source file. In a cross debug environment the debugger executes on the host development computer, and the application program runs on the target computer. The debugger reads the source and executable file on the host to get source code and symbol table information. The debugger communicates to an agent running on the target in order to examine variables and data structures and to control the process being debugged. The agent for GDB is the GDB Server and it is a lot smaller than GDB itself. GDB communicates to the GDB server either through a TCP/IP connection or through a raw serial connection.

The Linux kernel and device drivers cannot be debugged the same way as a user application. Setting a breakpoint in the kernel or otherwise stopping execution of a process while it is running in the kernel would cause the debugging agent to stop. Complete debug ability for the Linux kernel and device drivers is not supported in most desktop distributions of Linux. Some embedded distributions do support full kernel debugging. In this case GDB attaches to a debugging agent in the kernel usually through a low-level serial connection. This way the kernel or a device driver can be single stepped or breakpoints used even in interrupt routines. The only thing that cannot be debugged this way is the kernel debug agent itself.

There is more and more software available for Linux, not just open-source software, but third-party application programs and protocol stacks as well. The number of device drivers available for Linux is large and growing and the source code for Linux device drivers is almost always free. Linux has a good reputation for reliability on servers, and it has good network and file system performance.

Not only is the source code to Linux free, but runtimes of Linux are royalty free. For high volume embedded systems with tight per unit cost constraints, having no royalty payments for the embedded operating system is very attractive. It saves money that would go to royalty payments and eliminates the overhead for keeping track of the number of runtimes shipped.

The programming interface for Linux is UNIX. A major reason Linux has so much software available for it is that Linux uses the UNIX application programming interface as well as UNIX object and executable file formats. Much of the public domain and open source software of the last two decades has been written for UNIX and is now available for Linux. Also, third-party software developers have been providing products for flavors of UNIX for years such as AIX from IBM, Solaris from Sun Microsystems, and HP/UX from Hewlett-Packard. It was a simple matter for these software vendors to port their products to Linux. This also means that there are programmers experienced on writing UNIX software that already have the expertise to write Linux software.

Because of the popularity of Linux, there are already several software vendors that provide releases of Linux aimed at embedded systems. These vendors typically offer technical support programs for Linux that is geared toward embedded system development. This support includes bug fixes for Linux versions that may be years old and help with ports to embedded devices.

Is the operating system reliable enough and does it meet performance requirements?

Reliability and performance are two features of an operating system that usually cannot be augmented with application or library level software. Running reliable application software on an unreliable operating system simply makes the application unreliable. The reliability of an operating system is based on how well it was conceived, designed, and coded. Most embedded systems require a higher level of reliability than desktop systems. So choosing a reliable operating system is important. Of course, all operating system vendors claim that their operating systems are reliable, so an evaluation is often necessary to discern the facts. Performance can also be benchmarked. It is usually straightforward to determine if the operating system can meet the embedded systems throughput requirements. What is harder to determine is will the operating system meet the real-time requirements. Real-time performance can be benchmarked, but usually the difficulty lies with determining the real-time requirements of the embedded application. The real-time characteristics of system software can vary greatly as the system is put under load. All too often it is discovered that an operating system was the wrong choice because it could not meet real-time requirements that are not discovered until near the end of product development.

Linux is not a real-time operating system. The Linux kernel uses a fair share scheduling algorithm designed for time sharing, not a strict priority preemptive scheduler or some other scheduler suitable for real-time. But that is the least of the linux real-time problems. The Linux kernel is not preemptive nor reentrant by user processes. If one process is using a kernel facility no other process can execute until the process finishes or waits on something. This means the worst case response time of any real-time task running under the Linux kernel is longer than the time it takes to execute the longest stretch of kernel code before a wait. This time is unknown. The worst-case response time is further increased by memory cycle stealing by DMA devices and the CPU cycle stealing for the execution of interrupt handlers. There is no bound on the amount of CPU time used by interrupt handlers under Linux, and these run at a higher priority than any user process. Practically speaking, a user processes running on a very fast Pentium computer that is running a heavy load my take hundreds or even thousands of milliseconds to respond.

There are several ways to deal with the problem that Linux has poor real-time performance—ignore the problem, or run real-time applications under an RTOS with Linux itself as a separate task as in the case of RT-Linux, or run a Linux compatible RTOS kernel instead of the Linux kernel. The most popular approach is to ignore the problem. In most cases embedded developers have a problem identifying their real-time requirements at the beginning of a project anyway. This method my work if the application has extremely loose real-time requirements. If it is discovered that Linux is not meeting a real-time requirement then one of the other approaches can always be taken.

RT-Linux is a distribution of Linux that runs the Linux kernel as a task under a small real-time executive. Real-time tasks are run directly under the real-time executive and non-real-time tasks are run under Linux. This approach is not new. Years ago vendors marketed products to use this technique with commercial UNIX, and today several vendors use this approach to allow a computer running Windows NT to also execute real-time tasks. One of the advantages of the approach is that the real-time executive can be small and simple making it easier to verify its real-time performance. Another advantage is that the Linux kernel running non-real-time tasks is "standard" Linux so it is compatible with other Linux distributions and can be updated to a new release quickly. This approach also support hard real-time applications, ones where deadlines must be met absolutely every time.

There is certainly a set of real-time applications that can benefit from the RT-Linux approach. Machine control, process control, and instrumentation applications where the real-time application code is a small part of the entire application software should run under RT-Linux fine. Using RT-Linux is a much better approach than ignoring real-time requirements.

Some real-time applications do not fit the RT-Linux model well. Data acquisition to mass storage, telephone call setup or any requiring a real-time database, any application where the real-time part of the software is large or not easily separated from the code that needs a Linux environment. The problem is that tasks running under the real-time executive do not have access to the Linux facilities, device drivers, etc. And tasks running under the Linux kernel and can access the facilities are not real-time. As programmers apply the RT-Linux type dual kernel approach to applications where it is not a good fit the number of facilities in the real-time kernel will undoubtedly increased. At some point the real-time executive will no longer be small and simple, and as it adopts more of the facilities of the non-real-time kernel the question of whether the non-real-time kernel is required comes into doubt.

Another approach to the problem of running real-time applications on Linux is to replace the Linux kernel with a compatible kernel that has hard real-time performance characteristics. This may sound like sacrilege, but even in the RT-Linux case running real-time tasks under a Linux kernel is already abandoned. Also, with the advent of the Linux Standard Base (LSB), a binary interface standard for Linux, Linux becomes more than a technology but a platform for running applications. Any operating system that can meet the standard can run those Linux applications. The kernel is only a small part of a whole operating system. If just the Linux kernel is replaced and the libraries, utilities, and file structure remain the same, it's even easier to be compatible and conforming to the standard. The real-time kernel would have to support all the facilities of a Linux kernel while still remaining fully preemptive and reentrant. It would have to have a real-time scheduler and have bounded execution time for interrupt handlers.

The advantage of the replacement kernel approach is that there would not be a limit on the size or complexity of the real-time application code. Third party software not originally envisioned for real-time use could be invoked by real-time tasks. Programmers who understand Linux would already be experts in the real-time environment because the programming interface would be the same.

Despite the good reasons for choosing Linux for an embedded system or device, Linux does have some problems in the areas of licensing issues. Linux is not public domain software, it is licensed according to the Gnu Public License, which has a strict set of rules for use.

Linux is licensed under the terms of the Gnu Public License (GPL). Source code for any software under the GPL is free. Runtimes generated from source code under the GPL must also be free of charge. Any modifications to software under a GPL license automatically come under the GPL as well. You can't sell runtimes of GPL code for more than the cost of the distribution media and cost of distribution. If runtimes are distributed, the source code must be made public and readily available.

What does GPL mean to the embedded Linux developer? If you take the Linux kernel or any Linux utility and modify it, port it, or add features to it, you must post your source to the Internet or ship the source to anyone who asks for it. If you are not careful you may give up the rights to your proprietary software unintentionally. It is best to consult your attorney on all open-source copyright issues.

It is OK to ship Linux in a hardware device being sold. If any modifications to the Linux kernel, libraries, or utilities were required, the sources for the modified version must be made available free. Application programs that run under Linux may remain proprietary. Just be sure no part of the application is GPL (or other open source license) code. Device drivers can also be kept proprietary. They must be shown to be separate and distinct from the Linux kernel. This is most easily done if the device driver is written as a loadable kernel module. If it is linked as part of the base kernel it is in the gray area of whether it becomes GPL or not. If in doubt consult your intellectual property attorney.

Although Linux was not designed with embedded systems in mind, through careful configuration and by leveraging some of the work of the embedded Linux vendors, Linux can be successfully embedded in hardware products. Linux is emerging as more than just a technology but also a platform for embedded applications. Linux does have some problems in the areas of licensing issues, memory footprint, software development, headless and diskless operation, support for embedded processors, and real-time characteristics. Linux is not public-domain software, it is licensed according to the Gnu Public License, which has a strict set of rules for use. But because of its popularity, Linux will be considered for more and more embedded applications. When considering the main criteria for choosing an embedded system—availability of hardware and availability of third-party software, Linux has a lot going for it.