Thinking of using transparent GFP for long haul storage networks?

The Transparent Generic Framing Protocol (GFP-T) represents a new technique to extend storage area networks based on Fibre Channel to remote locations far beyond the data center. Although Transparent GFP-T extends the distance limitations of Fibre Channel for applications such as disaster recovery, it also poses some design problems engineers are just beginning to tackle.

The telecom world recently proposed GFP-T as a solution for routing datacom protocols across the wide area networks without needing to deploy new hardware. GFP-T eliminates the need for a Fibre-to-SONET router, often placed at an end user's location to link a Fibre Channel SAN with the WAN.

The Generic Framing Protocol (GFP) was originally developed to encapsulate IP packets or Ethernet MAC frames into Sonet. This efficient protocol simplifies the mapping of user data into Sonet frames in point-to-point, packet aggregation and resilient packet ring (RPR) applications. However, the difficulty of working on entire frames of variable length adds unnecessary cost and latency when addressing only point-to-point applications. GFP-T was introduced to eliminate this inefficiency as well as to address other applications.

GFP-T is a simple protocol that "transparently" encapsulates any 8B/10B encoded serial data into Sonet packets for transport across the WAN. GFP-T targets all the major datacom protocols such as ESCON, FICON, Fibre Channel, Gigabit Ethernet, HDTV and InfiniBand. Its goal is to provide a simple, easy-to-deploy method for point-to-point connections with minimal impact to the either end user or telco equipment. Early focus has been on multi-channel line cards that plug into the Sonet mapper/framers at the central office.

Using GFP-T each 8B/10B character of user data is decoded and re-encoded into a 64B/65B word. Eight words are grouped together into an octet with accompanying control and error flags. Eight octets are then assembled into a "superblock" which is scrambled and subject to cyclic redundancy checks.

The resulting packet is fully Sonet compatible. Port logic in the mapper/framer fills in the appropriate headers to route the packets correctly. GFP-T does not attempt to extract any useful information from the user data so the route through the MAN/WAN must be statically programmed with management software.

Routing data through the WAN benefits end users. Probably the most important feature of the WAN for these applications is virtual concatenation. This allows portions of a Sonet link's bandwidth to be allocated to certain data streams.

If a customer needs a remote 1.0625 Gbit/second Fibre Channel connection between San Jose and Denver, they pay for 1 Gbit of bandwidth even if the data is routed over a 10 Gbit OC-192 link within the WAN. The rest of the bandwidth on the OC-192 link can be allocated to other customers using virtual concatenation. Sonet availability, security, resiliency, error checking/correcting and flow control operate transparently to the end user.

Buffer, latency issues

While its benefits are clear, GFP-T also raises issues for Fibre Channel systems particularly in buffering and latency requirements that are just beginning to be addressed.

Most host bus adapters and switches today are optimized for use in data centers where links are short and buffer credit requirements are minimal. SAN designers must carefully review buffer sizes before buying Fibre Channel components to ensure that long distance links have adequate on-board or on-chip memory to keep their remote links powered adequately.

To maximize performance and to transfer the large blocks of the data found in typical storage traffic, systems typically transmit data fast enough to keep the media full of frames in transit. Fibre Channel links often operate at more than 80 percent of maximum bandwidth.

Fibre Channel uses a buffer credit mechanism to implement flow control. At initialization, nodes negotiate buffer size--usually 512 to 2 K Byte--and establish a value for the buffer credits per node based on available memory. After each frame is sent, the transmitting node deducts a buffer credit on behalf of the recipient.

Upon reaching zero, the transmitter halts transmission until it receives "Receiver Ready" acknowledgements (the R_RDY Ordered Sets) from the far end node. Each received R_RDY increments the transmitter's count of the receiver's buffer credit. In this manner, a Fibre Channel link is self-throttling and ensures that frames are not sent which will be dropped at the receiver. Unlike Ethernet, dropping frames is a serious offense in a Fibre Channel SAN.

Thus, SANs must be aware of each link's length and latency in order to ensure that adequate buffer credits are allocated to optimize performance. If a link travels 20 km within a metro network before being terminated by another Fibre Channel node, more buffer credits must be allocated to this link than to an link running just within the data center.

In order to establish the appropriate number of credits required, the round trip delay must be calculated. In purely optical links, the speed of the signal is approximately five nanoseconds per meter or 200 microseconds round trip for a 20 km link. The transmit time for a frame is the frame length (assume 2,172 characters for this example) times ten bits per characters times the bit period (942 picoseconds per bit at 1.0625 Gb/s) or approximately 20.5 microseconds.

At this rate, 10 buffer credits would be needed to keep the 20 km optical fiber stuffed with frames. A 100 km link would require approximately 50 credits or 100 Kbytes of buffers in each direction. A rule of thumb is that the link requires 1 credit for every 2 km of fiber.

Flow control

However, this calculation focuses only on latency due to long fiber optics but does not include the WAN itself which normally is on the order of tens of milliseconds. This increases the required buffer credits significantly.

At this time, a number of possibilities exist to address the flow control issues of remote storage applications.

First, vendors of Fibre Channel equipment may add additional buffering capabilities to their products. Second, an optimized Fibre Channel product might be developed which could sit between the SAN and the WAN that would add the necessary buffering capability. Third, the Fibre Channel community could develop a protocol variation, which is intended to operate over long haul distances. Fourth, a variation of GFP-T may be developed which targets Fibre Channel and FICON remote storage applications. Initial proposals are already being drawn up for this protocol.

The most popular solution in the short term will be provided by Fibre Channel to Sonet router/gateway vendors who will likely augmenting their products to add GFP-T support in order to increase performance. Their systems already contain large amounts of memory to address flow control issues in remote storage applications so their development effort centers on conversion to the new protocol.

GFP-T will out perform existing mappings onto Sonet due to its highly efficient transparent mapping. Once these Fibre Channel to GFP-T routers and gateways are available, SAN managers can cost effectively establish remote data centers with relative ease.

See related chart