Removing development bottlenecks through integration of tools and OS

By its very nature, embedded software development requires an extraordinary range of tools - everything from code editors and debuggers to tools for memory analysis, trace analysis, execution profiling, code coverage, version control, remote diagnostics, and more.

With so many tools, integration becomes critical. If a developer must learn a different user interface for each tool, or manually feed information from one tool to another, then time and effort is spent on mundane issues rather than on the task at hand - writing good software.In response to this need, some RTOS companies offer their own integrated development environments (IDEs). By all appearances, this is a good thing. After all, the term 'integrated' is meant to suggest that tools available for the IDE will work hand-in-hand to simplify, even eliminate, onerous programming tasks. Experience shows, however, that the actual level of integration can fall far short of this ideal.

For example:

* memory analysis tools that don't integrate with debugger tools

* editors that don't integrate with online help systems

* IDEs that don't support multi-language, multi-process, or multi-target debugging

* user interfaces whose appearance and behavior vary from tool to tool or from host to host.

What causes this tool 'disintegration'? In many cases, the problem occurs simply because tools vendors don't have a common framework to work from, a framework that provides a standard way for tools to talk to each other.

Thus, tools for one phase of development are unable to share information with tools for the preceding or subsequent phases. In the end, the developer must needlessly replicate or translate information as the project progresses - and the project itself progresses more slowly.

Inconsistent host support

RTOS vendors that offer conventional IDEs face another challenge: not all developers use the same development host. In fact, most IDEs must be ported to multiple hosts (Windows, Solaris, etc.), a process that is both time-consuming and expensive. It's no surprise, then, that RTOS vendors often focus their development work on one host, with the result that other hosts end up suffering from 'feature lag'.

Serialized development of hardware and software

These problems are further complicated by the conventional RTOSs that IDEs target. For instance, because most RTOSs can't support a full-featured desktop environment, developers can't choose to run the IDE on the RTOS itself. Instead, they must always develop on another OS, and then download and test their code on target hardware - the conventional cross-development model.

This can pose a serious delay in software development if, as often happens, the target hardware is unavailable until late in the project schedule. An RTOS that offers the choice of self-hosted development - and thereby allows the developer to do most testing directly on the development workstation - can bypass this bottleneck.

Extensive debugging

As a further complication, most RTOSs bind all system services (and, sometimes, all applications) to the kernel in a single memory address space. Developers must consequently learn a separate suite of kernel programming tools and techniques before they begin to develop device drivers and communication stacks. For this reason, development teams often end up hiring highly knowledgeable (and expensive) kernel 'gurus'.

Binding all system services to the kernel creates another drawback: a subtle programming error, such as a stray C pointer, in any driver or stack can corrupt the system without leaving a trace of what went wrong. Even with the best debugging tools, an error like this can sometimes take days or weeks to track down.

API lock-in

And, last, most RTOSs use a proprietary Application Program Interface (API). If for any reason a new project requires a different RTOS, developers must learn a completely different API - and a different IDE - then port any existing code they wish to reuse. Productivity suffers, as does return on code investment. Clearly, if software developers are to produce quality code quickly, they need both a better tools environment and a better OS architecture - one can't succeed without the other. Recognizing this, QNX Software Systems has developed a two-pronged approach to faster code development:

* QNX IDE - An standards-based platform that allows all development tools, even those provided by third parties, to share a common user interface and to share information throughout the development cycle.

* QNX RTOS - A POSIX-compliant microkernel OS that simplifies debugging and eliminates kernel programming by allowing every driver, communication stack, and application to run as a memory-protected process. Seamless tool-to-tool integration

The QNX IDE is built on the Eclipse platform, an open source framework that provides mechanisms and building blocks to simplify tool-to-tool integration - see information at end . The platform is quickly gaining acceptance in both the embedded and general software communities, with hundreds of thousands of downloads per month and the participation of over 150 tools vendors.

The IDE derives three key benefits from Eclipse: ease of integration, depth of integration, and consistency across multiple hosts. As for ease of integration, QNX has already demonstrated that third-party tools designed for eclipse can work seamlessly within the QNX IDE.

Fig 1: The QNX IDE provides consistent capabilities and a consistent user interface across all host platforms, including Windows, Solaris, and the QNX RTOS.

For instance, the Eclipse version of Rational ClearCase, a popular software change-management tool, can plug into QNX IDE without additional engineering effort.

As for depth of integration, tools that normally work in isolation can, in the QNX IDE, work together to reduce effort. For instance, the QNX memory analysis tool is tightly integrated with the debugger - when a memory allocation error is detected, the IDE can immediately switch to the debugger 'perspective', allowing the developer to quickly pinpoint the problem.

And as for consistency across development hosts, the Eclipse platform is Java-based and uses the standard widget toolkit (SWT) to integrate with the underlying window system, much like a native application.

As a result, the IDE doesn't have to be ported from one development host to another, which eliminates the problem of 'feature lag'. The net effect: all embedded developers on a project can share the same tools and source code, even if they don't all use the same development host.

Fig 2: Multiple target generation at the click of a button.

Faster development

Besides better tool-to-tool integration, QNX also offers the benefit of greater integration between tools and OS technology. Specifically, the QNX IDE can run on top of the QNX RTOS to provide self-hosted development:

Programmers can debug and test virtually all their software - including custom drivers, applications, and communication stacks - directly on an x86 workstation running QNX. No need to wait for target hardware. In fact, developers can perform this extensive testing on their x86 host even when the target hardware doesn't use an x86 processor!

This may seem surprising, since processors differ in how they handle memory alignment, atomic operations, byte ordering, and so on. But, taking advantage of the inherent hardware abstraction provided by the QNX RTOS's microkernel architecture, software engineers at have implemented functions and macros to address these issues, minimizing the need to rewrite and retest code for each CPU. Consequently, many QNX applications, drivers, and other services can be source-code identical across processors. In fact, developers can often build code for multiple processors simultaneously.

The IDE also supports cross development from Windows and Solaris hosts - regardless of the platform, developers can enjoy the same tools and user interface.

The IDE includes a full complement of tools for embedded development, from code creation to performance analysis. Moreover, these tools have an important thing in common: they're organized into perspectives.

In the IDE, a perspective brings together all the tools and information a programmer may need to perform a task - whether it's editing a file, debugging an executable, or generating a target image.

For instance, when you switch to the C/C++ perspective, you see the resources, library dependencies, and so on that comprise your project, along with the appropriate source file open in the IDE's text editor. Everything is immediately available so you can concentrate on the task at hand. And perspectives are persistent: If you switched to the debugger perspective, then back to the C/C++ perspective, everything in the C/C++ perspective would be as you left it.

Let's take a quick tour to see how the IDE's various tools work in concert to reduce development effort:

Integrated editing and debugging

To start, the QNX IDE has a C/C++ code development module that is integrated with the help system, debugger, and compiler - see figure 3.

Fig 3: Integrated editing and debugging

For instance, the code development module:

*displays context-sensitive help when you pause a cursor over a function in your code

* lets you set breakpoints and watchpoints for the debugger

*automatically highlights the offending source line when you click on a compile error.

Syntax aware, the code development module also provides other time-saving features, such as automatic code completion and error parsing.

As for the debugger, it offers a very powerful feature: the ability to debug multiple threads in multiple processes, even if the processes are distributed across multiple CPUs - it doesn't matter whether the CPUs are all on the same node or spread across multiple, loosely coupled nodes. As a result, faults in SMP applications and large cluster systems become much easier to pinpoint. The debugger can also dynamically attach to running processes, set process-wide breakpoints, and display the register sets and stacks of these entities. It also supports mixed source and assembly views.

System building

Building an OS image for an embedded system is typically a long and arduous process. To make the process both faster and easier, the QNX IDE includes a system builder plug-in.

The system builder provides a wizard that graphically walks the developer through the process of specifying the binaries, applications, drivers, and shared libraries to be included in the image. The system builder can also check for dependencies to ensure the developer includes all the files required. And, to minimize memory costs for the target, the builder provides a 'dietician' that shrinks libraries to include only the functionality required by the image.

Once a system image has been built, the system builder makes booting the target a snap by providing an integrated TFTP (Trivial File Transfer Protocol) server and a serial console. In addition to bootable images, the builder can help simplify the creation of custom flash or file system images.

Once a target image is generated, developers can take advantage of several analyzers to pinpoint timing conflicts, logic flaws, wasteful code, faulty resource synchronization, and a variety of other conditions that may affect performance.

First, there's the QNX profiler, which shows at a glance where a process spends its time, allowing developers to quickly locate 'sluggish' algorithms. The profiler can provide statistics for multiple processes and multiple targets, and performs both live and post-mortem profiling. Developers can also run instrumented code for gathering accurate call-count and call-pair information.

Note that developers can also perform post mortem analyses by using profiling information from the IDE and by examining process core files within the IDE's debugger. The QNX IDE can also display profiling information such as function histograms and call relationships in an intuitive, easy-to-comprehend view:

Memory analysis tool

Memory errors are the bane of most developers' existence. Thus, the IDE provides a memory analysis plug-in to help developers visualize the memory usage of their programs and detect common memory errors, such as buffer overruns and invalid de-allocations. Besides providing notification of memory error events, the memory analysis tool also works in concert with the debugger - when an error is found, the developer can activate the debugger perspective and track down the problem immediately.

System analysis toolkit

To help developers diagnose and fine-tune software behavior, the IDE also supports a system analysis plug-in. This tool works with the instrumented QNX kernel to capture the interactions of all components in the target system. The tool can time-stamp and display events such as process and thread creation, hardware interrupts, system calls, scheduling activities, IPC, and resource allocation. Using this tool, the developer can graphically pinpoint performance-degrading hotspots in an application, whether the application is running on a uniprocessor or a multiprocessor system.

To give developers a detailed view of the target system, whether it's in the lab and out in the field, the QNX IDE provides a target system information tool. This tool displays processor usage, memory usage, signal information, connections, register sets, states of processes and threads, and so on. A developer can retrieve this information on demand, or have the tool send periodic updates to show how the system is performing over time.

Managing large projects

The QNX IDE also comes with a product source manager for managing multiple, complex projects. This plug-in provides full control over make options, build order, project dependencies, and more. It also provides persistent views across development sessions. When you quit the IDE and restart it, resources remain configured as you left them - including window layouts, open files, and even breakpoints.

The project source manager also supports version control management (VCM), which provides a framework for plugging in a variety of source-code control systems. Developers can use the CVS plug-in that ships with the IDE or use a third-party plug-in, such as Rational ClearCase.

Additional eclipse tools

Because the QNX IDE is eclipse-based, it also inherits many additional tools built into the eclipse platform, such as internationalization, integrated help, and a local history that provides local resource management - great for dealing with those late-night development mistakes. The IDE also inherits a rich toolset for Java developers, including:

* Tools to create Java code - These include wizards for creating Java elements and a syntax-aware Java editor. The tools support building, refactoring, searching, etc.

*Java debugger - Can launch a Java VM in either run or debug mode, and can attach to a running VM. Supports dynamic class reloading, expression evaluation in the context of a stack frame, interactive Java code snippet evaluation, etc.

Eliminating the crash-reburn-reboot cycle

We've seen how the plug-in architecture of the QNX IDE eliminates the tool dis-integration of conventional IDEs.

Likewise, the microkernel architecture of the QNX RTOS eliminates many of the development bottlenecks associated with conventional RTOSs.

First of all, the QNX RTOS allows developers to modify embedded software without having to trudge through the time-consuming 'test-crash-debug-recompile-reburn-reboot-retest' cross-development cycle.

Based on a true microkernel, the QNX RTOS implements each application, driver, protocol stack, and OS module as a dynamically loadable, memory-protected POSIX process. As a result, a developer can dynamically load and unload any of these services on a target without having to restart the OS or download a whole new image.

In the QNX RTOS, the microkernel contains only a small set of core primitives. All other system services - drivers, file systems, protocol stacks - are provided by separate, memory-protected processes that can be stopped and started dynamically.

Even if a driver commits a memory fault (a fatal error for legacy OS kernels), the QNX RTOS remains running and can generate a dump file that identifies the exact instruction responsible for the error.

Using this debug information, the developer can quickly fix the problem, then simply recompile and reload the driver. No need for a system reset. (Of course, some other OSs can be dynamically modified to a degree, but they typically load modules into the kernel's memory space - when these modules crash or deadlock they take down the kernel.)

Sharing targets

Thanks to this dynamic, memory-protected architecture, multiple programmers can share one target system simultaneously while debugging a large number of modules. No one has to worry whether their modules will overwrite, or be overwritten, by anyone else's software. Companies can, as a result, reduce capital expenditures on target hardware - especially useful in applications where test hardware is either scarce or prohibitively expensive.

Fig 4: Process deep and system-wide memory analysis.

Faster development of system services

Because drivers and OS modules execute as memory-protected processes, the QNX RTOS also eliminates the complexity of kernel programming tools and techniques. Developers can implement drivers, communication stacks, custom file systems, and so on using the same, tightly integrated tools that the QNX IDE provides for developing standard applications.

To further simplify driver development, the QNX RTOS supports a driver framework that handles the default behaviors common to most devices. As a result, driver developers are free to concentrate on the low-level details specific to their device. And the driver framework is POSIX-based, so there are no arcane APIs to learn.

All this, combined with the ability to test changes in driver code without rebooting, provides unparalleled ease of driver development.

To simplify the development of distributed, multi-processor systems, QNX also provides network-transparent message passing: developers can partition an application across any number of nodes without having to write network-specific code. With QNX message passing, processes don't need to 'know' whether they're communicating with each other across the network; they simply pass messages to each other using standard POSIX functions, and the C library takes care of the rest.

Conventional RTOSs impose a double constraint on embedded development: a proprietary tools environment and a proprietary API. Together, these hamper tool integration, make porting difficult, increase training expenses, and serve to 'lock in' the customer.

Different approach

QNX takes a completely different, standards-based approach. First, it offers an IDE based on eclipse, which provides an open, and increasingly popular, framework for integrating tools from a variety of vendors. As a result, QNX developers will be able to leverage a larger choice of third-party tools tools that not only plug into the QNX IDE, but can interoperate with the rich suite of tools the IDE already provides. Second, the QNX RTOS is designed from the ground up to support POSIX APIs, which allow OEMs to leverage the large number of software developers with Linux/Unix experience, as well as an enormous pool of legacy source code.

To truly appreciate these benefits, it's important to remember that the QNX RTOS is much more scalable than conventional RTOSs.

With QNX, you can create one set of application binaries, then deploy them on a low-end uniprocessor device, on an SMP system, or across an immense cluster of loosely coupled processors. OEMs can, as a result, use not only the same OS, but the same developers, same APIs, same tested software, and, by extension, the same IDE across a much larger number of projects.

This level of scalability isn't merely desirable. Given the range of products many embedded manufacturers offer, it is, in fact, an immense commercial advantage - whether you consider development costs, time-to-market, product consistency, or, ultimately, customer satisfaction.

QNX IDE moves to beta phase

QNX Software Systems has started a beta program for its QNX integrated development environment (IDE), the first commercial IDE for embedded development based on the Eclipse platform.

Developers working with the beta release of the QNX IDE will have the advantage of a complete toolset for code development, debugging, project source management, memory analysis, and profiling - all tightly integrated under a single user interface.

The initial beta version of the tools include a C/C++ code developer that provides time-saving features like code completion, error parsing, and templates; a debugger that can debug multiple processes, languages, and targets simultaneously; a project source manager for handling complex multi-projects; performance analysis tools for memory usage and profiling; a flexible target agent architecture for debugging an embedded target over virtually any connection; and a system builder that lets programmers create embedded images with point-and-click simplicity.

Eclipse Consortium for open source development tools

Eclipse.org was set up in November last year by Borland, IBM, Merant, QNX Software Systems, Rational Software, RedHat, SuSE, and TogetherSoft. It is an open consortium of providers of development tools that manages the Eclipse Platform, which is available in open source under the Common Public License*.

The Eclipse Platform is an open source environment for creating, integrating and deploying application development tools for use across a broad range of computing technology. It provides a common set of services and establishes the framework, infrastructure and interactive workbench used by project developers to build application software and related elements. Through the Eclipse Platform, seamless integration of tools from several different vendors will be possible on Windows, Linux and QNX developer workstations.

The Eclipse Platform provides source code building blocks, plug-in frameworks and running examples that facilitate application tools development. A complete sample plug-in based integrated development environment for creating Java applications (JDT) is included. Code access and use is controlled through the Common Public License allows individuals to create derivative works with worldwide re-distribution rights that are royalty free.

The Eclipse Platform can be used to create and manage diverse objects like web site elements, process automation definitions, object models, image files, C++ programs, pervasive enterprise class Java applications and embedded technology. Written in the Java language, it comes with plug-in construction toolkits and examples, including a fully operational Java application development tools package. The platform implements a mechanism that discovers, loads and integrates the plug-ins developers need for manipulating and sharing project resources.

When the Eclipse Platform is launched, the user is presented with a workbench-based integrated development environment composed of the user interfaces of available plug-ins. * Some components of Eclipse may be governed by license terms other than the CPL.

Published in Embedded Systems May 2002