Switch fabric simulator simplifies QoS designs

Switch fabric simulator simplifies QoS designs
Developing a switch fabric with sufficient quality-of-service (QoS) capability for the sophisticated carrier and customer requires complex algorithms for prioritizing traffic and determining bandwidth allocation. Designers of switch and router software and firmware need access to these algorithms and their results in order to implement their own value-added features early in the design cycle.

Proceeding in this way reduces design risk and hugely improves time-to-market. However, most switch fabric vendors keep the details of their traffic-prioritization algorithms hidden — either in hardware, where they can't be accessed until a chip is soldered on a pcb, or beneath a "black-box" abstraction of the fabric, which itself is accessible only via a simple simulator. Either way, it's impossible for the system designer to get intuitive information about the fabric and to understand key real-world fabric behavior quickly.

The ability to simulate — in a cycle-accurate manner — system configuration values such as fabric cell size and number of traffic flows is extremely enlightening to system designers as they strive to find the optimal use of fabric bandwidth and prioritization capability.

And simulator performance is key. A slow simulator is extremely frustrating, and won't actually be used because it fails to achieve its core purpose: predicting the effects of changing variables on actual switch behavior in the real world, early in the switch/router design cycle.

This was the problem we faced in developing support software for our TeraChannel QoS-aware switch fabric. Our solution was the development of what we believe is the first cycle-accurate simulation model of a QoS-aware switch fabric.

Our design team designed the TeraChannel switch fabric with the aim of creating a data transport that combined bandwidth and intelligence.

Many fabrics are nothing more than "bandwidth on a board," or large point-to-point pipes within a switch. In such systems, the fabric has minimal scheduling capability and this crucial function is pushed outside to the packet-processing chips, which are not well-suited for it. The result is poor QoS support and an inability to support transaction-based billing, firewalls, encryption and the like. To take advantage of sophisticated network processor classification and queue-management abilities — and to implement QoS required for transaction-based systems — extra functionality is required in the fabric.

TeraChannel meets this need by integrating the bandwidth-allocation algorithms inside the fabric itself. Older shared-memory switches can accomplish this by the simple means of adjusting pointer position in the memory pool. Such switches are limited in bandwidth, though, and it is far more difficult to achieve useful bandwidth allocation in the input-queued crossbar designs needed for high-bandwidth chassis-based switches.

To deliver fine-grained control with input-queued crossbars, the bandwidth-allocation algorithms must be distributed across multiple devices in the fabric. An arbitration hierarchy combining a scheduling device, located on the central switching card, with an intelligent port device found on the line card allows bandwidth allocation by both priority and flow.

Implementing this hierarchy in TeraChannel required an understanding of complex behaviors and interactions among the bandwidth-allocation algorithms. This problem is tractable when each algorithm is analyzed in isolation from the others. However, when interactions among algorithms were added to the mix, the problem became too complex to deal with — beyond the capability of queuing theory and protocol verification tools.

The simplest and most flexible solution we first considered was to write an abstract model of TeraChannel in C++. Writing the model in this fashion made it possible to include detail only when and where it was required. The C++ model simulated quickly, enabling real-time what-if trials of new ideas in a full-size fabric — vital to ensuring that a given feature is scalable to large fabric port counts.

It was clear that running the model in this way gave valuable insight into TeraChannel behavior. It was also clear that customers would be helped immeasurably by having access to this capability.

But the internal TeraChannel model had by this time become convoluted, so it was abandoned to make way for the customer version, FabricSIM.

During the development and testing of the internal TeraChannel model, we learned several important lessons. First, simulation speed is vital. If the simulator is slow it is impossible to create useful statistical data. The model must be in a steady state for a significant percentage of the simulation time in order to get repeatable results. Since TeraChannel can support up to 5,136 virtual output queues, this is not a trivial requirement.

Second, in such complex designs as switch fabrics, interaction with the simulation increases understanding. It's difficult to gain intuitive understanding of fabric operation if the simulations are run in batch mode; that is, if test inputs are generated and then the simulator runs for hours before results may be analyzed.

Third, the model must reflect reality. If it doesn't, the model rapidly becomes misleading and, shortly thereafter, useless. Ensuring accuracy is one of the most difficult parts of the modeling process. The internal TeraChannel model was accurate enough to be used as a gold-standard reference during FabricSIM development.

To satisfy all of these requirements, we felt that the production version of the FabricSIM simulator had to be fundamentally different than the internal TeraChannel model. The internal model had representations for each device in the TeraChannel switch fabric. By contrast, FabricSIM represents the fabric as a whole, which allows the use of abstract data types and event-based simulation. Switching from timing-based to event-based simulation increased speed by an order of magnitude.

With the increase in speed, we were able to build an interactive interface that displays result graphs and tables even as values are being changed.

The core of the simulator contains all of the algorithmic functionality of the TeraChannel fabric and all of the necessary statistics-gathering functions. Every useful aspect of the operation of the fabric is recorded and can be passed to the outside world.

We also incorporate traffic generators, which support varying loads and packet-size distributions.