Linux is on the NPU control plane

Linux is on the NPU control plane
In the last 24 months, with the economic woes faced by merchant fabric suppliers and the advent of integrated network processors (NPUs) such as Intel IXP and IBM PowerNP, Linux is finding more application opportunities on the control and management planes of NPU-based networking equipment.

The control plane constitutes the logical and physical interfaces that do not themselves transport network data but that set up, monitor and control the activities of silicon that does. In current NPU designs, an embedded operating system such as Linux executes on the control-plane CPU, governing the operation and performing housekeeping activities on behalf of packet engines, the specialized silicon that actually manipulates Internet Protocol packets. The connection between the control-plane CPU and the data/forwarding plane packet engines can be as intimate as a local-bus connection to a coprocessor, as straightforward as standard PCI, or as arm's length as Ethernet or application-specific network fabric.

In the case of a local bus or of PCI, the control processor can treat the forwarding/data plane processor(s) as specialized peripherals that present, superficially, regular memory-mapped interfaces. In the case of a networked connection between the planes, the control processor must employ a split device-driver design. First, the fabric interface requires a generic driver (for Ethernet, asynchronous transfer mode, etc.), over which specialized command packets are sent to control the packet processor. The richness of embedded Linux networking (Ethernet and ATM are both readily available in Open Source) gives the OS a marked advantage over proprietary off-the-shelf and in-house kernels for which developing specialized fabric or even Ethernet drivers can represent a significant additional investment.

Off-the-shelf NPUs strive to integrate high networking functionality closely tied to the control CPU with local connections. First-generation designs, however, have suffered from bandwidth limitations at the control point and in the data plane — data packets can face significant bottlenecks in moving both to and from the control plane, as well as among parallel packet engines within the NPU. Moreover, network processors with powerful and complex packet-engine sets have proven to be notoriously difficult to program. NPU vendors today address these performance issues in their silicon roadmaps and with new generations of libraries and tool kits.

Linux/NPU apps

Device OEMs are deploying Linux and NPUs together in an ever-increasing array of applications, specifically in network infrastructure and client devices.

A key area for Linux on NPU platforms is small-office, home office (SOHO) designs such as access points, home gateways and digital media servers where the economic realities of consumer electronics demand high functionality and low cost.

Many of these same designs previously contained legacy real-time operating systems (RTOS) platforms. This raises the question of real-time performance in embedded Linux. Many control-plane applications, while needing throughput, have no hard real-time requirements (despite prior choice of a real-time operating system). Newer, NPU-based designs present even less stringent response needs to the control CPU, with NPUs subsuming as they do performance-intensive operations in specialized silicon. And when hard deadlines must be met in the control plane — as with routing and bridging to certain isochronous interfaces employed in consumer electronics applications — multiple paths exist for reaching real-time responsiveness in Linux, including the MontaVista Software preemption technology now present in the Linux 2.6 kernel.

In larger, more complex IP-based systems, including central office and infrastructure applications involving IP telephony, high-speed access and switching, network processors are finding their way into blade designs with two or more levels of control and management hardware and software above them. Overarching 5 and 6 Nines requirements for many of these systems demand high levels of redundancy on the data and control planes. In such systems, the NPU-based blades and their Linux-based controllers are treated as lower-level, almost atomic interfaces, subject to fail-over, with highly available and fault-tolerant management tiers presiding over their operation.

These higher-level management-plane processors are also increasingly running Linux, albeit carrier-grade versions of Linux, as defined by the Open Source Development Labs Inc. (OSDL, Beaverton, Ore.) and implemented by MontaVista Software, Intel Corp. (Santa Clara, Calif.) and others. These management-plane CPUs administer available resources (such as booting the blades and allocating load to them), respond to hardware and software faults and provide application bridges to enterprise-type systems (often also running Linux) such as billing systems and subscriber databases.

As with any multicore microprocessor, network processors present multipath development and debugging challenges. These challenges are threefold:

Developing and deploying code and packet-handling algorithms that execute on the NPU itself;

Crafting interfaces between the NPU and the control-plane CPU;

Developing the control-plane application.

To a control-plane processor running embedded Linux, the NPU initially appears as a large, complex and unsupported peripheral. At a minimum, startup kernel code or driver initialization code will map a combination of shared memory and registers that represent the NPU interface into kernel or user memory, such that drivers and applications can go about their business of interacting with the NPU. That mapping may then terminate in user-level APIs, explicit driver APIs, or it may end up subsuming the bottom of a split protocol stack, facilitating a socket or streams-based abstraction.

Device OEMs look to Linux platform providers and to a growing ecosystem of independent software vendors (ISVs) to supplement their own efforts in building NPU-based network appliances. The building blocks supplied by ISVs fall into several groupings:

Packet-handling algorithms, interface bridges and NPU-specific development tools for the network engines on the data plane (packet-engine code assembler, simulators and performance analyzers);

Control and management-plane application elements-Simple Network Management Protocol (SNMP) agents and management information bases (MIBs), Web control interfaces, etc., protocol stacks, routing daemons, control and data-plane interface libraries;

"Tall" stacks for particular applications, such as home gateways and security appliances;

Other system and application development tools for Linux.

Companies with NPU interface and IP offerings for embedded Linux include Ashley Laurent, Hughes, IP Infusion, LVL7 Systems, NextHop Technologies and Motorola's Netplane Systems, which offers stacks and libraries. Silicon vendors themselves, such as Intel and IBM, as well as ISVs such as Teja Technologies, offer development tools that complement the architecture-focused tool sets for particular NPU silicon.

The tall stacks support and implement as much as 90 percent of an NPU application and appear either as add-on products from platform vendors or integrated offerings from boutique software vendors. Such stacks appeal most to second- and third-tier device OEMs and to original design manufacturers (ODMs) whose sole differentiation may be brand or price point; leading networking OEMs, however, will continue to invest in rolling their own NPU applications for the foreseeable future.

Since many NPUs also sport integrated JTAG interfaces to both the control processor and the packet engines, developers in some cases can use common hardware-debug interfaces from such vendors as Abatron AG, Embedded Performance, Wind River International and others to both debug control-plane operations and download and interface directly to NPU packet engines.

Perhaps the most fruitful marriage of NPUs and embedded Linux comes from combining Intel's IXP425 family with Linux-based access and network storage application stacks. A good example of this synergy is the D-Link Storage Media Central Home Drive. D-Link and dozens of other (mostly Asian) device OEMs leverage the integration of an XScale control/applications CPU with high-performance network processor functionality. In the case of the D-Link product, the Taiwan-based OEM built on this hardware base, used the integrated packet-processing facilities to provide fast and secure Ethernet and wireless connectivity. D-Link combined that hardware platform with the routing, multiprotocol, and multiformat storage capabilities native to Linux to provide a plug-and-play consumer network storage solution. Other recent deployments of Linux-based NPU applications include the ThingMagic RFID system.

On the horizon this year are dozens of applications for home, SOHO, enterprise and infrastructure. These include access points, attached/wireless storage solutions, virtual private networks (VPNs), security appliances, digital subscriber line access multiplexers (DSLAMs) and xDSL line cards, networked printers, residential gateways, IP/PBXs, SOHO routers, residential gateways, as well as core/edge applications supporting OC-3 through OC-192 high-speed networks.

Bill Weinberg is director, strategic planning at MontaVista Software Inc. (Sunnyvale, Calif.).

See related chart