Harnessing the power of high speed interconnects with software

As high-speed serial interconnects come of age, the distribution of processing power across multiple, independent, processing units has become easier and much more commonplace. These large bandwidth pipes, which are now available to interconnect processing devices, offer more options for the system designer to create ever more powerful and flexible designs. However, with power comes the need to control that power, and in such systems there is a need to provide mechanisms to control the bandwidth between processing units in order to best leverage the distributed processing that can exist.

Many benefits exist to designing a system with multiple processing units interconnected via high speed interconnects. By interconnecting the processing units it is possible to share the processing power of the system across the devices. Certain processing units can be dedicated to specific tasks and act as a service resource to the remainder of the system, while other devices may have similar processing function deployed across multiple like-units. The interconnection of these devices allows for the design and deployment of highly scalable systems that can provide an overall highly fault-tolerant system.

Whether the processing units are separate, remotely located devices interconnected via external fiber optic or copper cabling, or if they are blades within a single enclosure such as a PICMG chassis, the same challenges exist regarding how to distribute, control and manage the processing power that is enabled by the high speed interconnects. These challenges are expanded when the system architecture is one that embodies heterogeneous processing devices consisting of differing CPU architectures or perhaps operating systems. The configuration and management of these multiple processing devices as a unified system is perhaps one of the greatest challenges facing the system architect.

In order to fully leverage the power of all aspects of the interconnected, interacting processing system, a system controller must be able to be coordinate, control and modify the system during operation much as a conductor controls the various sections of an orchestra throughout a performance.

A system must be devised to allow the configuration and control of the multiple processing elements. In order to configure each unique device, these processing elements must first be known to the system. This may be accomplished by the means of static configuration to the system controller or by the means of some type of dynamic system discovery protocol. Numerous standards-based protocols exist to enable the configuration across these devices. If the processing devices are to be treated as loosely coupled units, one viable approach is to utilize IP application protocols such as SNMP (Simple Network Management Protocol) as a standard way to interface to the devices. By assigning each device a unique IP address, they may be individually managed from the system controller or a remote entity.

The use of SNMP as the communication and configuration mechanism allows for needed information to be defined in well structured Management Information Bases, or MIBs, and enables the reuse of a mature management infrastructure. If the processing devices are being treated as a more tightly coupled processing unit, other techniques can be leveraged to provide inter-process communication between the units and the system controller. In this scenario, SNMP could still be leveraged with the system controller responding to external SNMP management and then performing proxy access to the individual processing devices.

By having a consistent mechanism for the configuration and management of each processing unit it would then be possible for each processing unit to report its capabilities and characteristics so that the system controller can leverage the various devices for the purposes of distributing the overall system processing load. This ability introduces an additional challenge of providing load balancing and service provisioning within the system. The system controller must be able to determine the current overall processing load of each individual element as well as the pending demands for processing and allocate request to the appropriate processing devices.

Flow control

One technique to maximize the systems overall performance would be to allow each processing element to perform unique, but coordinated functions. In this model the processed data will be flowing from one processing element to another in succession. This can bring about a situation where one processing element may not be available to receive the new incoming data as it may still be processing the previous information. As such, mechanisms for flow control must be built into all aspects of the system from the serial interconnect itself to the control system coordinating the processing across the devices.

If the processing system is to be made scalable and fault tolerant to some level, it is likely that some type of switching fabric will be needed to allowed for mesh interconnection of these devices. With data flows moving from multiple processing elements destined to various locations within the system there arises the possibility for contention of the interconnect resources.

As such, there is a need to factor in capabilities for prioritizing the traffic with the interconnect. Once again techniques proven in the Ethernet / IP world can be leveraged. If the interconnect fabric is leveraging Gigabit or 10 Gigabit Ethernet, traffic can be assigned priority via the 8 levels of priority defined in the 802.3ac VLAN Tag header. If the traffic is being sent over IP, then additional techniques such as DiffServ or other Quality of Service protocols can be leveraged to provide various levels of control on this traffic.

While there exist a number of proven techniques and protocols that can be applied to distributed processing systems leveraging high speed interconnect techniques, there exist few "standard" methods for designing such systems. Thus far discussion has centered around the higher level actions that that are needed at a system level to provide control and coordination of the overall processing system. However, as earlier discussed these systems are likely to be heterogeneous systems that contain various CPU architectures and operating systems.

As such, the actual mechanisms and implementations for the control communication between devices will vary across processing elements, however they will all require a certain base set of functional needs. Problems as simple as byte ordering driven by different CPU architectures need to be handled in a consistent manner.

Unified front

A unified software architecture that presents a singular control and configuration plane can solve many of the challenges presented here. A singular control and configuration plane need not mean that all processing of the control and configuration activity occur on a single processor. It may in fact be distributed across multiple processing elements. An architecture that segments the control and configuration plane across the system controller and the individual processing elements can ease the distribution and being to enable the redundancy needed in high availability systems.

In distributed processing systems that leverage serial interconnect switching fabrics the system can be viewed as one having a control / management plane (the system controller) interconnected via the switching fabric to the data / forwarding plane (one or more processing devices). The processing devices are responsible for the actual processing and manipulation of the data while the system controller is responsible for the overall coordination of the system.

Furthermore, the system controller is responsible for the management of the collective system resources, handling of system errors such as processing device failure or other system failures that would affect the availability of the system, and the configuration and external management of the collective system. By segmenting the system architecture into such a view the system can be setup to leverage a common software base for the system controller as well as a common base set of software that interacts with the rest of the system for the processing devices. The processing devices would then provide the necessary software that adapts these standard system interfaces to the unique nature of its particular hardware configuration and capabilities.

While modern high speed interconnects are enabling the development of highly powerful, scalable and redundant processing systems there remain a number of system architectural challenges that must be addressed to maximize the power and flexibility of such a system. The need to provide a consistent configuration and control mechanism across all of the devices in the system is paramount in order to present a unified, functional system.

Leveraging existing protocols from the Ethernet and IP arena can provide a useful toolkit in dealing with contention that may arise for access to system resources interconnecting the multiple processing systems.

Finally, treating the distributed system as one with a well defined control / configuration plane along with a data / forwarding plane will provide for a consistent system software architecture that maximizes the efficiency of the system.