Design considerations for 10-Gbit Fibre Channel

The 10-Gbit Fibre Channel (10GFC) specification has been stable for about a year, and revision 3.4 is out for the third and final ANSI public review (see www.t11.org, document03-076v1.pdf). The signal and timing characteristics for both the 3.1875-Gbit/s and 4.25-Gbit/s data rates are defined in FC-PI-2 (see the "Drafts" section in www.t11.org). Now that 10GFC using four lanes running at 3.1875 Gbits/s has been stabilized, the spec using 4.25-Gbit/s data rates is expected to be finalized in April 2003.

The 10GFC multilane connection will be primarily used between fabric switches and later between large disk subsystems. The single-lane 4GFC will be a future speed enhancement for current 1GFC and 2GFC dual-ported disk drives.

The ANSI Fibre Channel and IEEE Ethernet committees have shared the pain of development by using common media, encoding, control logic, timing and signal specifications for 1 Gbit and above. Fibre Channel did the majority of the development at 1 Gbit, with Ethernet doing the majority for 10 Gbit.

At the 10-Gbit level, the cooperation was expanded to allow the use of the same common components. That reduced the cost and time of development. The result was a common physical-coding sublayer (PCS) at almost double the production volume and a reduced price.

In terms of interfaces, designers defined a 10-bit parallel Gigabit Media Independent Interface (GMII) for Gigabit Ethernet and extended it to a 40-bit parallel XGMII interface at 10 Gbits. Each 10-bit section comprises 8 bits of data, 1 bit to flag special k-characters and 1 bit to flag errors.

The next trick was to make four serial streams running at 3.1875 GHz look like one stream at 12.75 GHz. The 10 Gbits/s of actual data transferred in each direction is a result of the 8-bit to 10-bit (8b/10b) encoding method used for the serial stream (12.75 GHz x 80 percent = 10.2 GHz). We define this 10-Gbit Attachment Unit Interface (Xaui) as four lanes of differential encoded serial data over copper.

The XGMII-to-Xaui transceiver is also referred to as the XGMII extender sublayer (XGXS). The transmit and receive paths are completely separate to provide full-duplex

operation with 2 Gbytes/second of total usable bandwidth.

There are three forms of Fibre Channel controls: data packets, primitive signals and primitive sequences. Ethernet only uses data packets and primitive sequences.

The XGMII-to-Xaui transceiver is also referred to as the XGMII extender sublayer (XGXS). The transmit and receive paths are completely separate to provide full-duplex operation with 2 Gbytes/s of total usable bandwidth.

During normal operation, data packets are paced by buffer-to-buffer credit primitive signals sent during the gap between frames. These provide reliable full-bandwidth data transfers without the data retransmission and partial-bandwidth characteristics of Ethernet. Primitive sequences are used during circuit initialization to establish the point-to-point connection.

For those who choose to implement the PCS/XGXS layer, there are a few details to observe. Most of these are defined within clauses 9 through 12 of 10GFC. The transmit and receive state machines are summarized quite well in the informative Annex D of 10GFC.

One area that may require extra care centers on the alignment of the four lanes. The method used by 10GFC and 10GbE differs from that used by Infiniband.

Reducing EMI

First, the transmit and receive state machines have no idea of frames or primitive signals. XGXS only knows data, and idles with a special case for primitive sequences. An illegal k-character (k07) is used on the XGMII to indicate an interframe gap (IFG). During this time the transmit state machine inserts a random pattern of k-characters to reduce EMI effects: k28.3 for alignment (A), k28.5 for comma (K) and k28.0 for skip (R).

All compliant implementations are required to forward the following k-characters: k27.7, for start frame (lane-0 only, S); k29.7, for terminate frame (T); k30.7; for any error (E); k28.4, for primitive sequence (lane-0, Q); k28.2, for Fibre Channel primitive signal (lane-0, P); and k28.1, k28.6 and k23.7, which are important for future expansion. K28.7 is not supported by 64B/66B devices and generates an unaligned comma character when two characters are adjacent.

The receive Xaui data is synchronous with the remote transmitter. An elasticity FIFO (EFIFO) is used to speed-match this data with the local system clock, which is normally also used by the local Xaui transmitter. Skips are inserted into the EFIFO when the remote transmitter is slower or deleted if the remote transmitter is faster. It is recommended that the EFIFO automatically recover from overflow or underflow and mark the occurrence with a column of k30.7 (IB uses PAD, k23.7).

The 8b/10b coding uses a combination of disparity and encoding to find errors and provide a dc-balanced self-clocking serial stream. Think of this as 2 bits of encoded parity added to 8-bit data. K-characters always terminate disparity errors. Invalid characters are those whose disparity and encoding do not match a predefined equation. The current character disparity is used to verify the next character. Since an illegal character depends on the previous character, it is recommended that both be replaced by k30.7, referred to as a check-end function in the 10GbE spec.

The 10GFC/10GbE alignment method is known as static alignment. With static alignment, any insert or delete happens at the same time for all four lanes. That keeps the data aligned for as long as the link is operational. Bit errors are ignored by requiring the alignment k-characters to be misaligned three times before realignment is required. Normally the alignment unit comes before the EFIFO; they may also be combined.

Infiniband, on the other hand, uses dynamic lane alignment, based on the number of Skip k-characters per lane. Infiniband does not require the Skip to be inserted or deleted at the same time. The Skips are grouped together during the interframe gap and surrounded by random data to reduce EMI. For this reason, the alignment unit must constantly be realigned on the last Skip k-character received. This method allows the elasticity FIFO to come before the alignment FIFO.

For longer distances, repeaters are used to reestablish the exact timing and to insert and delete Skip k-characters at the same time for all lanes. A repeater can be built by simply connecting the XGMII receive data and clock signals to the transmit data and clock signals, and providing a crystal reference clock.

Based on my work as co-editor and facilitator for the 10GFC spec and in implementing Fibre Channel, Infiniband and PCI-Express cores, I have proposed extending Fibre Channel to 20 and 40GFC using multiple 10GFC Xaui components and connections. A dynamic lane alignment function can be added above the XGMII interface to present a single 20GFC or 40GFC data stream. (See "20GFC/40GFC using four-lane Xaui," in document 03-069v0.pdf at www.t11.com.)

See related chart