This article is an introduction to the application of serial switched PCI in systems designs. Serial Switched PCI inherently provides key benefits of increased flexibility, scalability, and reliability, and eliminates many limitations inherent in bus-based architectures. The article will introduce the architectural freedoms resulting from these benefits, as well as the class of service capabilities available from serial switched PCI technologies. In particular, this article will demonstrate how the application of serial switched PCI expands the capabilities of existing PCI based designs. Several real world applications illustrate the benefits and ease of use of StarFabric, the only shipping serial switched PCI technology.
System Interconnects
Typical computing systems operate with a number of interconnects, designed to optimize specific attributes for system sub-sections. Local, proprietary parallel buses connect CPU, chipset, and cache/memory; these buses are very short and designed to maximize speed. A mezzanine bus is commonly used to connect devices within the system chassis. The mezzanine bus has also been a shared parallel connection that connects graphics, storage, and communications modules to the system chipset. It is designed with goals of performance and flexibility in the number and type of attached devices. The building-level LAN (a serial connection designed for moderate distance and performance over low cost cabling) is commonly used outside the system chassis—between systems, clients, and networked storage. It is a serial connection running over low cost cabling, at lower performance levels and moderate distance. Both the mezzanine and LAN are conventionally built to industry standard architectures to take advantage of the number of standards compliant devices.
In many designs over the last 10 years, the most popular mezzanine bus has been the PCI bus. PCI has evolved over the years from base PCI (32b/33MHz) to faster and wider PCI implementations to increase performance, but PCI is inherently limited by its shared parallel bus structure. A shared bus architecture is susceptible to being halted by a single failure and is limited in scalability by the demands placed on devices to transmit and receive signals to and from all connected devices. This places severe limits on the physical length of the bus and the number of devices within a system. Further, the bandwidth available on this bus is only available to a pair of devices at a given time.
PCI to PCI bridges provide a solution that increases the number of potential connections, but at the expense of latency. Cabling of the PCI signals to extend the bus to multiple chassis is restricted to the range of 1 meter. Again, these limitations conspire to limit performance and scalability. Despite these limitations, PCI has been an overwhelming success, with an installed base of systems and peripherals that far overshadow any other interconnect scheme. It has been the enabling technology of the modern, plug-and-play, upgradeable PC and embedded system. A technology that could truly extend the PCI bus and provide new features and benefits while maintaining compatibility with existing system infrastructure would provide enormous benefits to the computing world.
Figure 1: Evolution of system interconnect |
Switched Fabrics
Switched Fabrics consist of multiple point-to-point connections between nodes that are interconnected through a network of switches. This "fabric" can consist of a nearly limitless number of switches and connections. The topology of the fabric is not fixed, as in a parallel bus-based system. There are a minimum number of switches and connections that are required to route a given number of nodes, but the topology can be augmented to provide redundant paths and switches, increasing performance and reliability.
A fabric can be built on paths of any width. For reasons of economics and signal integrity, it is advantageous to reduce the number of traces or wires between nodes, therefore reducing the number of paths that need to be laid out, matched in characteristics, and built. A successful approach requires that the data is transmitted in a serialized fashion through the fabric and is de-serialized at the point where it leaves the fabric. The advantage of the serial approach over parallel operation lies in the fact that the speed at which the signal can be serialized/de-serialized and transmitted will far outstrip over time the ability to build synchronized wide parallel buses of any useful width and length. The addition of the ability to build in redundancy and multiple paths increases the advantage of the serial approach.
StarFabric
StarFabric is a serial switched fabric technology with fully specified physical, electrical, and protocol layers. The technology is incorporated into high performance switch chips and bridge chips that interface to external buses. StarFabric links operate electrically at 622Mbps with LVDS signaling. The pairs are aggregated 4x to provide 2.5Gbps. Separate paths for incoming and outgoing traffic are provided, and signals can be routed on a standard PCB or through cable such as CAT5E. StarFabric on CAT5E cabling can span distances to 13 meters.
The StarFabric frame is passed through the fabric with routing information encoded in the frame header of the serialized data payload. There are several routing methods defined for StarFabric. The simplest method, address routing, offers the benefit that it is totally compatible with PCI, using existing hardware, OS, configuration utilities, BIOS, and drivers. It is a simple extension to existing PCI based systems.
StarFabric also features path routing, which provides advanced features including multiple classes of service for real time and priority data. This feature enables the transport of time-sensitive data over StarFabric alongside the general traffic, and not through a separate bus, with its associated costs and system complexity. Path routing also enables high availability features like fault detection, automatic failover, and hot plug. Path routing does require some additional software to take advantage of these features, but it provides significant benefits over legacy PCI mode.
Figure 2: PCI expansion to additional chassis |
PCI Expansion
The simplest PCI expansion can be accomplished with a PCI-StarFabric Bridge connecting a primary PCI bus to a secondary (potentially remote) PCI bus through a second PCI-StarFabric Bridge. Since the StarGen PCI-StarFabric Bridge (the SG2010) has two StarFabric ports, it has the ability to expand to two secondary buses in a point-to-point manner (Figure 2). In order to expand to more buses, a switch is required. The SG1010 is a six port StarFabric Switch. Its uses greatly expand the scope of the fabric. A single SG1010 can connect six separate PCI buses, or multiple switches can be used to build redundant paths for increased reliability and performance (Figure 3). This expansion of the bus is useful for increasing the number of peripheral devices (such as storage controllers) in a system, but also to implement multi-processing systems with distributed processing and control.
Figure 3: StarFabric switch scales number of endpoints |
Using address routing, StarFabric provides a very straightforward path to expand PCI based systems to new levels of performance and size, and offers the potential for connection to remote resources, with a robust and transparent fabric. This is an attractive proposition in many applications where PCI or CompactPCI is the incumbent interconnect. The following example applications will illustrate how this technology is being used in real-world systems to increase their performance and scope and to extend the usefulness and life of existing designs.
Storage Expansion
Storage needs in all types of systems are increasing, and disks have moved out of the system box to remote box(es). The movement of these large quantities of data presents a challenge. StarFabric can scale from simple storage expansion using address routing, to enterprise ready highly available solutions using path routing. Figure 4 illustrates several possible system solutions.
Figure 4: Storage expansion with StarFabric |
Image Processing
Modern imaging systems have much in common between medical, military, weather, or air control applications. System operators at the host computer control the scanning equipment and display the graphical output generated from DSP units operating on large quantities of scanned data. The scanning system can be MRI, PET, ultrasound, radar, or sonar to acquire the image of the patient, weather system/airspace, and so on. Often these functions are remote from each other, and sophisticated storage of the data is required. StarFabric allows greater freedom to expand and locate these system blocks, without the use of proprietary custom interconnect technology. In addition, StarFabric increases the bandwidth available to support adding DSP, storage, or I/O resources.
Figure 5: Image processing system benefits from StarFabric serial switched PCI |
Communications Control Plane
Many communications systems have been designed with either PCI or Ethernet as the control plane, setting up the connections on the switch and resource cards. PCI is limited in the number of cards supported on the bus without incurring crippling latency issues. Ethernet can handle a much greater number of connections but requires resources at each card to handle the Ethernet overhead. As Ethernet has evolved from 10Mbs to 100Mbs to 1Gbs, the increasing costs and decreasing efficiency and performance are a serious concern. StarFabric control plane solutions hold advantages over either PCI or Ethernet, providing bandwidth and physical scalability, low latency, low line card overhead, low cost, and increased reliability and high availability.
Figure 6: StarFabric control plane with redundant controllers |
Summary
Switched Serial PCI is a very attractive interconnect to the system designer requiring greater levels of flexibility, reliability, and scalability than are available from today's PCI mezzanine bus. Whether the need is in distance extension, bandwidth scaling, connection scaling, or advanced routing and reliability, switched serial PCI provides a significant upgrade path to today's PCI-based design. It provides a low cost, high performance alternative to Ethernet in other applications. StarFabric is a low-cost and robust implementation of serial switched PCI with bridges and switches that connect over PCB traces or CAT5E cabling. System designers can implement StarFabric technology in existing PCI systems with un-modified PCI software. StarFabric is industry tested, and shipping in production systems today.
About the Author
Patrick Hart is a Marketing Consultant at StarGen. Prior to StarGen, Pat was with API Networks, Digital Equipment Corp and Texas Instruments. He has 20 years of experience in marketing and design in the electronics and computer industry. He received a BSEE from Marquette University.
For additional information about StarFabric, visit the StarGen
Web site.
