Serial Rapid IO: Speeding Up Control-Plane Designs

Serial Rapid IO: Speeding Up Control-Plane Designs
Curbing bottlenecks in today's communication equipment designs has clearly become one of the biggest challenges for the design engineering community. And in these architectures, the backplane has become a key focal point.

With more and more blades being crammed into racks and higher data rates traversing boxes, building an efficient data plane that can efficiently and quickly move data has become vital. Any slow downs in this plane can dramatically hinder system performance and, in the long run, the overall operation of a communication network.

Clearly, the control plane has become one of the key choke points in the overall backplane design. With that in mind, a host of companies has developed a serial version of the RapidIO specification, dubbed Serial RapidIO, to improve overall backplane performance.

While the parallel version of RapidIO has been around for some time and is well understood, the serial version is relatively new and not well grasped in the design community. To help out, let's take a deeper look at the spec and its impact on the design process.

Changing Tradition

Traditionally, networking and telecom equipment designers have tackled control plane design using PCI architectures. With the adoption of the CompactPCI (CPCI) specification several years back, PCI solidified its presence even further as a key ingredient in control plane developments.

But PCI brings with it a host of problems that cause huge headaches and bottlenecks in today's equipment designs. While architectures based on PCI are elegant and simple, they are becoming increasingly limited in bandwidth and scalability. Additionally, the ability to propagate signals on a PCI bus with many loads and connectors is becoming more and more difficult as bus speeds increase, to the point where PCI-X at 133 MHz has essentially become a point-to-point protocol.

Furthermore, since only one device can transmit on the bus at any moment, concurrent conversations between multiple devices is not possible. While this approach may have been OK in systems operating up to OC-48 speeds, as designers begin the push to OC-192 levels and beyond, this single transaction approach creates huge traffic slowdowns in a system architecture.

Finally, a wide parallel bus consumes too many pins on the system backplane. These pins are a scarce resource and are needed for the data-path where the speed and throughput requirements of the data justifies the number of pins.

So, without a doubt, PCI is a problem and new answers are needed (see "HyperTransport Fabrics: The Next wave in PCI?" Communication Systems Design, November 2001, www.csdmag.com). And that answer is coming in the form of RapidIO.

Why RapidIO?

Serial RapidIO provides an ideal solution that overcomes the limitations of parallel buses for the control path. RapidIO provides link bandwidth that is comparable to PCI, with the benefit of scaleability beyond PCI through the use of a switched architecture. Also, because the bus arbitration problem is eliminated, a high degree of concurrent peer-to-peer interconnection is possible. Serial RapidIO only consumes four pins on a backplane, providing a very efficient, reliable interconnect (Figure 1).

Click here for Figure 1

Figure 1: Diagram of a typical control path built around the RapidIO architecture.

In addition, many of the systems that are being designed today require the so-called five 9s availability (99.999% uptime). PCI as a backplane bus lacks the error handling and fault isolation that fault tolerant systems require. RapidIO, on the other hand, provides wire-speed error detection and correction in hardware, enabling true five 9s availability.

The Protocol

To understand RapidIO, you must first understand the layers that make it up. Specifically, the RapidIO protocol is divided into three layers - the logical layer, the transport layer, and the physical layer (PHY) (Figure 2).

Click here for Figure 2

Figure 2: Breakdown of the three layers of the serial RapidIO specification.

The logical layer contains the logical abstraction for communication. This includes three types of I/Os - memory-mapped, global shared memory, and message passing.

Of the three I/Os, the memory-mapped I/O is probably the most significant in the serial RapidIO structure. The term memory-mapped I/O is used to describe a scheme where hardware registers respond to addresses in memory space. These registers provide windows in memory space to the I/O functions such as an input FIFO, for example.

From a software perspective, this is easy to use. A pointer or data structure is constructed that points to the location in memory for the I/O device. A simple read or write to that memory location is all that is required to perform the I/O operation.

The global shared memory specification defines a number of coherent transactions. These include, READ, READ_TO_OWN, IO_READ, CASTOUT, FLUSH, IKILL, DKILL. The message passing specification defines how messages are sent. They are associated with a single endpoint port. Messages are stored in a register at the destination port, and can have a doorbell function.

The Transport Layer

The transport layer defines how data packets are transmitted between end-points through the fabric. Latency is important in embedded systems. As a result, routing is simplified in RapidIO switches.

Most of the intelligence required to route traffic is contained in the endpoints. The source and destination addresses for all endpoints are contained in the header of a packet. These fields can be either 8 or 16 b. Initial RapidIO implementations will use 8-b source and destination fields. This means that first generation RapidIO products will allow for up to 256 endpoint routes in a system. This level of scalability goes beyond the needs of almost any large system.

The PHY defines the physical interconnectivity. Today, there are two PHY standards in RapidIO: the 8/16LP parallel and 1/4LP serial specification. Designers can choose between either PHY. But for purposes of this discussion, we will look at the serial PHY.

In addition to the three layers, the Serial RapidIO spec calls two types of transactions: packets and control symbols. Packets are used to transmit logical layer and transport layer data. Control symbols are used to transmit PHY handshake information. This includes things like packet IDs, buffer status for flow control, and physical errors like cyclic redundancy check (CRC) violations.

The parallel specification consists of one clock, one frame signal, and eight or 16 data signals. Serializing this requires a parallel-to-serial conversion of the data, and encoding the frame and the clock. The serialized data is 8B/10B encoded. This ensures that the clock signal can be extracted using conventional PLL-based clock recovery techniques. This is the same subcoding layer that exists in current Gigabit Ethernet or Fibre Channel system architectures. The frame signal is encoded through the use of special control symbols. The control symbols, which are capable of delineating packets, include start-of-packet, end-of-packet, stomp, restart-from-retry, and link-request

Control symbols can appear at any time in a RapidIO data-stream. They can be inserted in the middle of a data packet. Figure 3 shows how the serial physical layer is used to wrap a RapidIO packet and how control symbols can interleave within a packet.

Click here for Figure 3

Figure 3: Control symbols can appear at any time in a RapidIO data-stream. As shown here, control symbols are inserted in the middle of a data packet.

The serial PHY provides several features that are also present in the parallel specification, although the details of their implementation is different by virtue of being a serial link. These features include guaranteed message delivery, flow control, PHY priorities, error management, deadlock prevention hardware, and transaction ordering. The interoperability specification ensures that the transaction ordering rules for PCI map to RapidIO priorities such that PCI ordering rules can be maintained across a RapidIO fabric.

Simplified Programming

In order for any I/O technology to be successful, one point is clear. If a simple programming model is not included, the I/O technology will be useless.

Just look at PCI. While PCI has its problems, it also provided some interesting advantages in communication networking designs. The biggest, however, is that PCI allowed designers to develop systems using a relatively simple memory-mapped I/O programming model.

Serial RapidIO tries to mimic this trend by using a memory-mapped programming model, which is similar to the programming model used in PCI, is engineered to keep software overhead to a minimum, and is simple to use.

Some companies have recently advocated using Ethernet for the control plane. It is well understood, pin efficient, and cost effective. It has reached commodity status by many accounts. However, Ethernet comes encumbered with a large heavy-weight protocol stack.

A fully assembled TCP/IP packet to encapsulate a simple 32-b register write is woefully inefficient, requiring the transmission of 14 times more data than is strictly required. Furthermore, messaging models like TCP/IP are burdened with the additional latency of the required handshake. Software at the receiving end has to unpack the packet using messaging software and additional latency is incurred. In the end, it is 97% less efficient than a simple memory write which takes a handful of clock cycles in PCI. Memory-mapped I/O is simple, robust, and provides all of the flexibility required to build the most complex systems.

Link Bandwidth

The bandwidth available for control path is also important. PCI provides reasonable bandwidth and relatively low latency for many control path applications. However, it is very pin hungry. In contrast, Serial RapidIO provides comparable bandwidth and latency at much lower pin counts. Table 1 below shows a comparison of the interface widths and bandwidths that are available in both PCI and Serial RapidIO.

Table1: Available Widths and Bandwidths for PCI and Serial RapidIO

Link	Signal pin count	Link bandwidth	Aggregate bandwidth
PCI 32/33	54	133 MB/s	133 MB/s
PCI 64/66	93	533 MB/s	533 MB/s
PCI 32/66	54	266 MB/s	266 MB/s
PCI-X 64/133	94	1 GB/s	1 GB/s
RapidIO 1x 1.25	4	125 MB/s (1 Gb)	n x 125 MB/s*
RapidIO 1x 2.5	4	250 MB/s	n x 250 MB/s*
RapidIO 1x 3.125	4	312 MB/s	n x 312 MB/s*
RapidIO 4x 1.25	16	500 MB/s	n x 500 MB/s*
RapidIO 4x 2.5	16	1 GB/s	n x 1 GB/s*
RapidIO 4x 3.125	16	1.25 GB/s (10 Gb)	n x 1.25 GB/s*

*Switched non-blocking architecture provides full link bandwidth with no congestion.

There are several data rates that map well to other existing protocols. The 1.25-Gb x 1-b serial rate maps well to carrying Gigabit Ethernet traffic. The 3.125-Gb x 4-b link maps well to OC-192 and 10 Gigabit Ethernet. The 3.125-Gb x 1-b serial link maps well to OC-48. Matching I/O link bandwidths is important when considering how to aggregate traffic in networking equipment.

The pin and performance advantages of serial RapidIO are clear. Serial RapidIO is the obvious choice for building pin-frugal, high-bandwidth control path interfaces.

Silicon from leading semiconductor vendors will support Serial RapidIO for a variety of designs. This will include CPU vendors, interconnect solutions, I/O chip sets, FPGAs, and DSPs to name a few. This breadth of support for the RapidIO standard together with the features that support high reliability real-time applications make RapidIO a compelling choice for anyone wishing to build high-performance systems.

About the Author

Victor Menasce is a director and lead architect at Tundra Semiconductor where he is responsible for new product definition. Prior to joining Tundra Semiconductor in 1997, Victor was responsible for development of three generations of processors for voice switches at Nortel Networks. Victor received a Bachelors in Electrical Engineering in 1986 from DalTech in Halifax, Nova Scotia and can be reached at victor@tundra.com.