Merging ATCA, PCI Express Opens Next-Gen Backplane Designs

Merging ATCA, PCI Express Opens Next-Gen Backplane Designs
Despite the availability of popular and affordable open-architecture backplanes such as CompactPCI, many members of the communication infrastructure community still create their systems using proprietary backplane solutions. This has been especially true in the telecommunications market, where there is a clear need for standard implementations that allow more highly integrated blades to be part of complex chassis-based systems.

The PICMG group has a strong history of creating standard backplanes using readily available technology. New efforts are underway within the organization to merge the new PCI Express specification with the upcoming Advanced Telecom Computing Architecture (ATCA). Once complete, this combination will provide an open backplane that can more easily scale to meet current and future communication equipment design requirements.

The ATCA Architecture
CompactPCI has been successful in a wide range of communications-based applications, but has not seen that same level of success in the telecommunications industry. The primary restrictions that have prevented a more complete penetration into this important category are form factor and power/cooling distribution. Complex telecommunication systems rely upon large numbers of ports, and sophisticated processing of the ingress and egress data. In order to accommodate this type of system, the chassis must provide the space and power infrastructure to allow large numbers of high dissipation components on each line card.

ATCA will provide an 8U board form factor that is 280-mm deep. Each board in an ATCA system can accommodate up to four standard PMC mezzanine modules. The size of the board itself is an advantage over previous standardized solutions. There is space to include network processing engines, physical translation of optical and electrical signals, large banks of DSP processors, and copious memory for each subsystem.

The 8U form factor is particularly suitable for 12U high chassis, three of which will fit in a standard 42U high telecom cabinet while leaving enough space for power distribution, alarm panels, and more. At 1.2 inches, board-to-board spacing for ATCA is sufficient to accommodate large heat sinks, DC/DC converters, and DIMM modules while providing adequate airflow for reliable cooling.

The intended mechanical environment for ATCA is the 600 x 600-mm cabinet specified by ETSI, although 19- and 23-in. rackmount frames are also specified. With a 600 mm-deep cabinet, the ATCA mechanicals leave 90 mm of space between a board and the front access door for cabling and an air plenum, plus 75 mm in front of the rear access door for metal cabling and another air plenum. Also in accord with telecom equipment practices, ATCA opted for dual redundant -48V power supplies and specifies high-current connectors.

The ATCA specification permits 200 W of power dissipation per slot, and boards taking up multiple slots may dissipate power in multiple 200 W increments. It is anticipated that advanced cooling techniques in the future may be able to accommodate boards with a higher power draw. Rear-panel I/O modules, in turn, are allowed 10W per slot.

The physical aspects of ATCA are an enabler, and provide a foundation that allows enough space and power to drive modern heterogeneous telecommunications systems. But the actual transmission of data through the backplane happens within the fabric section of the ATCA standard.

Integrating Fabrics to the ATCA
ATCA is agnostic when it comes to fabric topology, supporting a variety of implementations from stars to redundant dual stars all the way up to full meshes. The goal of the standard is to allow the user to choose the least expensive solution that provides the performance and redundancy necessary.

A straightforward star, with a switch board having a direct connection to every node board in the fabric, is the simplest and least expensive topological variation, but it lacks redundancy, a key reliability issue for mission-critical high-end systems. The switch board represents a single point of potential failure.

A dual-star extended fabric topology brings that redundancy to ATCA (Figure 1), dovetailing nicely with the other built-in redundancies of the spec for power supplies, for example, and for system management. In a dual-star architecture, two switch boards are employed. Each of these boards offers a dedicated link to every fabric node, typically with one of the switch boards serving as a backup in case the primary switch board fails. This is a common and relatively straightforward mechanism to provide redundancy.

Click here for Figure 1
Figure 1: ATCA's dual star topology offers built-in reliability, but two slots are required for switch boards, making them unavailable for functional nodes.

The most flexible and powerful topology is a full mesh (Figure 2), where discrete switch boards are eliminated altogether. Instead, each node board contains its own integrated switches and each board connects directly to every other board in the fabric. While this requires extra real estate and cost for node boards, it allows all slots to be used for functional modules rather than for switches.

Click here for Figure 2
Figure 1: The full-mesh topology is the ultimate in redundancy but nodes require a built-in switch, adding to their cost.

The benefits of eliminating fabric boards are significant. The end customer makes his money by offering ports to the outside world. The fabric boards are by their nature overhead. By eliminating the fabric slots, the full mesh enables the system to offer usable ports in every slot, thus maximizing the revenue possible from any number of slots. The full mesh offers this benefit while also offering the most redundancy possible. Since every node is connected to every other node with a direct link, any single point of failure is completely removed.

Merging PCI Express and ATCA
Specifications for the extended fabrics of ATCA are at various stages of development. PCI Express is specified under the PICMG 3.4 committee, where approximately 30 companies are actively participating in its definition, which is expected to be completed in Q1 2003.

There are two aspects of the process of taking PCI Express and mapping it onto the basic ATCA system. The first is the feature set of the PCI Express standard itself, which was explained in a previous article. Features such as traffic classes matched to virtual channels to allow quality of service; peer-to-peer ability to allow smart nodes to send and receive information without routing through a host bottleneck; host failover ability to ensure that even centralized resources are duplicated; are all of particular interest in the communications space. Given the highly reliability demands of telecommunications customers, these take on even more importance in that market.

One interesting aspect of ATCA that is not offered in any other sub-spec is the ability to combine base and advanced switching nodes without needing to have special bridging. The PCI Express base specification offers a serialized, point-to-point, scalable, state-of-the-art backplane system, but does so while remaining entirely backward compatible with the large (and still growing) infrastructure of PCI software. This is a tremendous boost to the new standard, and it ensures that there will be a wide variety of base compliant hardware as early as next year.

The advanced switching addition to the PCI Express includes features such as peer-to-peer node interaction, multicast, and multi-host failover. But in the process of providing these features there is no possibility of remaining backward compatible with a base of software that has no conception of these features.

The good news is that the base and advanced switching protocols are physically and link layer compatible. As such, the PICMG 3.4 specification can and will include the ability to combine systems that make use of the earlier base hardware and software and the newer, more powerful advanced switching hardware in a single system. Since the two PCI Express standards are physically compatible, the fabric card can sense which version is connected to the node and react accordingly. This ability to offer mixed systems without the need for physical bridging is a unique and powerful feature that is offered in no other ATCA specification.

PCI Express has an additional feature not available from any of the other sub-specifications of ATCA, where mixed systems of any kind can be created through the combination of physical bridging and encapsulation. Rather than reformatting the contents, PCI Express can simply treat another protocol as data and add the appropriate PCI Express headers, stripping them off again at the destination. This allows OEMs to take advantage of the performance, routing efficiencies, and standardized features of PCI Express without force fitting a solution or forcing OEMs to give up their legacy protocols.

Ethernet, for example, will be around for a long time, and fibre channel is well entrenched in SANs. In VoIP systems, data may come in over SONET, then be packetized into IP and then into Ethernet. Taking advantage of protocol encapsulation, computer and communications OEMs can carry over much of their previous efforts to new generations of equipment without software interference and reduce cost while gaining backward compatibility.

PCI Express Specific Issues for PICMG 3.4
It has been explained that most of the factors that are necessary to map PCI Express onto the ATCA chassis are either standardized or at least outlined in the base PICMG 3.0 spec. The primary protocol-specific issues that the PICMG 3.4 subcommittee is currently addressing are fabric pin out allocation for zone 2, electronic keying, and signal integrity assurance.

Pin out allocation standardizes the data connection, and allows for PICMG 3.4 boards to be easily integrated into a PICMG 3.4 chassis. This is a straightforward matter, and involves identifying the signal flow for each topology (star, dual star, mesh). The signals in a PICMG 3.0 chassis travel along links that are identified as Channels.

Electronic keying uses the system information provided by boards and backplanes to ensure compatible link technologies exist between any two system boards prior to enabling the fabric interface for any and all links (Channels) within the backplane. To ensure that accidental insertion of non PCI Express cards do not damage the chassis, PICMG has defined a procedure that identifies PICMG 3.4 compliant cards. The fabric manager will not assert main power until it receives the proper link type from the card. PICMG 3.4 allows for both PCI Express base and PCI Express advanced switching to interoperate within the chassis, which provides design flexibility.

Signal integrity is a more complicated matter. While signal integrity is largely handled by the PICMG 3.0 specification, the PICMG 3.4 committee does need to ensure that the 2.5 GHz signals that are defined in the PCI Express specification can travel reliably along the ATCA connectors and backplanes.

PCI Express utilizes LVDS to provide high speed, noise resistant connections that are easy to route. Detailed simulation and validation must be budgeted to ensure a successful implementation.

Two of the more significant design issues are designing for signal loss and crosstalk. Let's look at both in detail, starting with signal loss.

PCI Express allows a total of 13.2 dB loss between transmitter and receiver. Designers must account for not only the resistive, dielectric, and skin effects of the connectors and vias, but also trace lengths, trace geometries, and the number of layers utilized in the PCB. PCI Express allows up to 20 inches between components using standard FR4 and high volume/low cost connections. However designers must consider other issues when designing their board. For example, designers should limit the bends within the traces, and each bend should be less than 45 degrees. Vias should similarly be limited to two on each traces for the motherboard.

Crosstalk can be minimized by keeping pair to pair spacing larger than the pacing between a pair. If possible, transmit and receive differential pairs should route alternately on the same PCB layers.

The advantage to bundling PCI Express with ATCA is that it provides a standard electrical, mechanical and datacommunication implementation. This allows design engineers to focus their efforts on their board implementation and relegates their backplane interconnection to ensuring compliance to the ATCA specification.

Wrap Up
PICMG 3.0 ATCA alleviates much of the issues that drive companies toward proprietary systems for next-generation telecommunications equipment. When coupled with the robust, well supported, and standardized features of PICMG 3.4 PCI Express, vendors will enable not only a cost effective solution, but also one that is scalable, with higher performance and greater reliability and flexibility.

About the Author
Danny Chi is a product manager at PLX Technology. He holds a BSEE from the University of California, Los Angeles and an MBA from Carnegie Mellon University. Danny can be reached at dchi@plxtech.com.