Manning Up for 10 Gigabit Ethernet

With this demand for bandwidth, the metropolitan area network (MAN), access network, and transport network have become key concerns in the carrier and networking equipment developer communities. As the core of the Internet moves to 10-Gbps operation and beyond, there is also a push to bring more bandwidth and higher speed operation to MAN, transport, and access equipment. That push has forced the design community to explore new technologies in the wired, optical, and wireless areas for equipment that deliver bigger pipes.

So far, 10 Gigabit Ethernet is one technology that is garnering a great deal of attention as a new means for solving the bandwidth demands of communication designers. Although initially seen as an upgrade to existing in-building LAN environments or as an uplink or trunk link for existing Gigabit Ethernet equipment, 10 Gigabit Ethernet has also emerged as a key solution for building metropolitan optical access and transport networks that deliver 10-Gbps data rates and beyond.

Existing metro structures

To better illustrate the impact of 10 Gigabit Ethernet in the MAN and access world, consider a typical metro optical ring structure. Existing metro and access ring architectures are based on a SONET/ SDH add/drop multiplexing scheme. A hierarchical ring structure is used to aggregate traffic from local central offices. Faster rings then carry this data to Internet point-of-presence (POP) locations for processing. The traffic is routed back on the rings for local destinations, or sent to long-haul transmission equipment if destined for another distant location. Provisioning and management of metro and optical rings is very time consuming.

Internet POPs are typically comprised of edge and core routers that are connected by a POP switch. The core/edge router architecture is required due processing limits and lack of ports on core routers. This segmentation provides an efficient method for aggregating low-rate traffic by using edge routers with many high-level features and core routers that focus on high-speed routing, with fewer features.

Data enters a POP from metro rings and moves into edge routers. The data is either routed locally or sent to core routers for processing. For redundancy, edge routers are dual-homed or connected to more than one core router. A POP with many edge routers consumes expensive core router interfaces and is difficult to connect physically. Layer 2 or layer 3 POP switches are used to simplify the physical connections between edge and core routers. POP switches also provide aggregation of edge router traffic to minimize the number of core router interfaces used for intra-POP connections.

10 Gigabit Ethernet may have a significant impact on this Internet structure in both architecture and cost. In the POP, 10 Gigabit Ethernet switches allow current architectures to scale to higher bandwidths while further reducing the number of expensive router ports required for intra-POP connections. New metro architectures utilizing meshed and partially meshed 10 Gigabit Ethernet switches aim to reduce infrastructure costs and simplify management and provisioning of services.

Preparing for the long haul

Typically the long-haul network is comprised primarily of SONET/SDH and optical multiplexing equipment. In this network structure, voice, video, and data are transported across transport links using SONET/SDH framing. Data is transported over these networks via ATM (that is, IP over ATM) or packet over SONET/SDH (IP over SONET/SDH). 10 Gigabit Ethernet technology has not only found a home in the metro and access equipment market, but has also emerged as a viable technology for long-haul optical systems. In these applications, 10 Gigabit Ethernet competes head to head with packet-over SONET/SDH (POS) for the uplink between core routers and transport equipment.

Internet traffic from enterprise networks predominantly originates from Ethernet LAN segments. In a POP environment, routers, POP switches, and server farms process Ethernet frames as well. POS requires the conversion of Ethernet frames carrying IP into POS packets (point-to-point protocol [PPP] and high-level data link control [HDLC] encapsulated IP). The use of 10 Gigabit Ethernet instead of POS reduces the amount of packet processing and corresponding system cost by preserving the protocol format of the data. This data can then be sent directly across the entire SONET/SDH transport network, eliminating a whole layer of equipment, along with the associated provisioning and management overhead, while reducing end-to-end traffic delays.

A tale of two PHYs

The IEEE 802.3 is the standards body responsible for Ethernet specifications. As of late, the main focus of this standards group is the development of 10 Gigabit Ethernet standards (IEEE P802.3ae) as the next level of evolution of Ethernet to address high-bandwidth applications.

In order to service two broad network applications, the IEEE is defining a 10 Gigabit Ethernet WAN physical (PHY) layer and 10 Gigabit Ethernet LAN PHY. The LAN PHY is intended to maximize the data rate to 10 Gbps, while the WAN PHY is rate compatible with the existing OC-192 (9.95328 Gbps) WAN infrastructure.

The IEEE P802.3ae has developed a draft specification based on the components listed in Table 1. This table shows the LAN PHY and WAN PHY components for serial and 4-l (4 wavelength) optical modules. The main differences between the LAN and WAN PHYs are in the data rate, coding sublayer, physical medium dependent (PMD) interface, and link management.

Table 1: 10-Gigabit Ethernet Draft Standard Components
Stack layer	10 Gigabit Ethernet LAN PHY		10 Gigabit Ethernet WAN PHY
	Serial	WWDM	Serial	WWDM
MAC data rate	10.0 Gbps	10.0 Gbps	10.0/9.58 Gbps*	10.0/9.58 Gbps*
Optional MAC/PCS I/f	XGMII	XGMII	XGMII	XGMII
Optional XGM II extender	XAUI	XAUI	XAUI	XAUI
PCS	64B/66B	8B/10B	64B/66B SONET framing x7+x6+1	64B/66B SONET framing x7+x6+1
PMA interface	XSBI	XAUI	XSBI	XSBI
PMA/PMD	1550-nm DFB 1310-nm FP	4 1400-nm CWDM	1550-nm DFB 1310-nm FP or VCSEL 850-nm VCSEL	4 1300-nm CWDM
Line rate	10.3 Gbps	4 x 3.125 Gbps 1400-nm CWDM	9.953 Gbps	4 x 2.5 Gbps

The data rate and frame structure for the WAN PHY were specifically engineered to match current SONET/SDH WAN and optical networking data rates. This was done so that 10 Gigabit Ethernet traffic could be format-and rate-compatible with existing SONET/SDH and optical transport infrastructure. Optimizations have been made to simplify clocking and management of traditional SONET for 10 Gigabit Ethernet applications to reduce cost and complexity. Note, however, that a 10 Gigabit Ethernet WAN PHY cannot be plugged directly into standard SONET equipment. A layer 1 (path) relay function is required to perform the electrical, optical, and overhead conversion between SONET equipment and 10 Gigabit Ethernet equipment.

Match that rate

Rate matching is required to accommodate a rate of 10 Gbps at the media access control (MAC) and the WAN PHY running at 9.953-Gbps line rate. During rate matching, the MAC will insert extra spaces (idle characters) between Ethernet frames to bring the effective data rate down to match that of the WAN PHY, while still running at a 10-Gbps clock rate. The WAN PHY will then delete these idle characters to allow the Ethernet frame stream to be packed into a SONET/SDH payload at 9.58464 Gbps. The WAN PHY therefore suffers a small (4%) reduction of throughput as compared to the LAN PHY.

The LAN PHY data rate is chosen to operate at 10 Gbps to optimize for throughput. In order to facilitate longer reach applications, additional fiber management capability has been added. The line rate of the LAN PHY depends on the coding scheme employed. The serial LAN PHY uses 64B/66B coding, while in applications using 4-l optics, 8B/10B is used. This results in data rates of 10.3 Gbps and four lanes of 3.125 Gbps respectively.

The WAN PHY employs a basic SONET frame and scrambling to transport Ethernet data. The 64B/66B code characters generated from the same coding scheme used by the LAN PHY are encapsulated in SONET frames rather than being directly fed to the optics. Frame delineation within the received SONET payload is accomplished by recognizing valid 64B/66B data blocks. Table 2 examines each coding scheme along with its benefits and limitations. The IEEE P802.3ae also set distance objectives to ensure that cost-effective PHY solutions exist for short reach, intermediate reach, and long reach applications. Table 3 provides an overview of the optical modules being defined to support the various distance objectives.

Table 2: Physical Coding Sublayer Summary
Coding	Use	Benefits	Limitations
8B/10B	Coding scheme for the XAUI interface and the WWDM LAN PHY, when using serial optics, and the WAN PHY for delineation of Ethernet frame data across four separate bit lanes when using 4- optics modules.	Simple and proven technology as 8B/10B was used for Gigabit Ethernet data transmission.	25% coding overhead
64B/66B	Coding scheme for the LAN PHY, when using serial optics, and the WAN PHY for delineation of Ethernet frame data within the SONET/SDH payload. 66B/66B is a framed scrambler adding -2 frame bits per 64 scrambled bits using x58+ x19+1 polynomial.	Smaller coding overhead compared to 8B/10B (3%) which allows serial optics to be used in either LAN or WAN PHY applications. Optical module range from 9.953 to 12.5 Gbps too wide for cost-effective production. Where 9.953 to 10/3 Gbps is a reasonable operating range.	64B/66B is a new coding scheme. Analysis shows however, that 64B/66B should be more robus than 8B/10B. Actual performance to be determined.
SONET/SDH framing and scrambling	SONET/SDH frame is used as the frame structure for the 10 Gigabit Ethernet WAN PHY. 64B/66B encoded Ethernet frames are carried in the payload of the SONET/SDH scrambler is used to ensure sufficient 1s density in the transmitted data stream.	SONET/SDH frame enables 10 Gigabit Ethernet WAN PHY compatibility with existing SONET/SDH-based metro and transport equipment. Also, SONET/SDH framing limits the line rate for compatibility with existing photonic metro and transport equipment. Scrambling the data allows the receiver to recover the clock and also serves to reduce electro-magnetic interference. SONET/SDH overhead bytes provide extensive network and fiber plant management.	SONET/SDH framing and overhead bytes reduce data throughput by 4%.

Table 3: Optical Module Solutions to Meet IEEE Distance Objectives
Optical module	Distance/media	Description
1550-nm serial	2 m - 40 km+ over single mode fiber	For ling reach applications, 40 km and beyond, 1550-nm DFB optical modules are used. These optical modules are being deployed today for OC-192 applications.
1310-nm serial	2 m - 10 km+ over single mode fiber	Intermediate reach mode fiber applications 2 -10 km use 1310-nm FP serial optical modules. Fastest time to market for WAN PHY by leveraging existing OC-192 optical modules.
1300-nm	2 m - 10 km+ over single mode fiber	Utilize lower rate optics for time-to-market and simplify manufacturing process to reduce cost.
850-nm serial	2 - 300m over 50µ(2000 MHz-km) MMF 2 - 86m over 50µ(500 MHz-km) MMF 2 - 69m over 50µ(400 MHz-km) MMF 2 - 35m over 50µ(200 MHz-km) MMF 2 - 28m over 50µ(200 MHz-km) MMF	To address the short reach 10 Gigabit Ethernet applications, a number of companies are developing 850-nm serial VCSEL-based optics.

There are two optional interfaces for connection to optical modules, XSBI and XAUI. These interfaces are described with their benefits and limitations in Table 4.

Table 4: Optical Module Interfaces
Coding	Use	Benefits	Limitations
XAUI	To extend the XGMII interface up tp 20" and allow transmission across connectors. Also used to connect a 10GMAC to 4- optics modules in LAN PHY mode. 8B/10B code recognition used to align data across the 4-b lanes. Note that the coding overhead of 8B/10B does not seriously impact 4- optical modules as each lane is operating at 1/4 of the aggregate data path rate (1/4 of the gain per lane can be easily managed with 3.125 Gbps).	Allows greater separation of the 10GMAC and optical modules for applications where these devices are on different system cards. Allows LAN PHY to leverage 4- optics.	Currently, intermediate and long reach optical modules do no support this interface.
XSBI	LAN and WAN PHY applications where serial optics are used. XSBI is a modified version of the SFI-4 OC-192 interface developed at the Optical Interworking Forum (OIF). The operating range of the SFI-4 interface was increased for XSBI to include both LAN and WAN applications. SFI-4 16 at 622 Mbps XSBI, 16 b at 622 Mbps (WAN) or 645 Mbps (LAN)	Leverages the OIF SFI-4 interface, which is widely deployed in currrent OC-192 optics. The XSBI is the only interface that is common to both the LAN and WAN PHY. Note that the IEEE defines a scheme to map 16-b data to 4 b for use with 4- optical modules. Intermediate and long reach optical modules use this interface. The WAN PHY could use existing optics for time-to-market.	Requires more pins and power than XAUI.

Applications

Great debates are raging in the industry as to where each 10 Gigabit Ethernet PHY type fits. Those considering the LAN PHY are looking to maximize data throughput and leverage the volumes from the enterprise deployment. Those who prefer the WAN PHY are looking to leverage the WAN infrastructure compatibility for metro and transport applications.

Management is a key issue in choosing a LAN versus a WAN PHY. Some applications may require both SONET and LAN management, where others will focus on only one, depending on existing experience and tools. The WAN PHY has both SONET/SDH (using an optimized set of overhead bytes used) and traditional LAN management where the LAN PHY only uses LAN management.

On the Ethernet transport front, multiplexing equipment would typically deploy 10 Gigabit Ethernet WAN PHYs, as the rate and frame type are compatible with existing systems. The use of LAN PHY cards would require buffering and control to perform rate matching. Transport multiplexers are focused on high-speed long-haul transport of data. To minimize end-to-end system delay a minimal amount of buffering is utilized.

Rate matching (that is, from 10.0 to 9.953 Gbps) requires buffering to ensure that flow control mechanisms activate before data is lost. Adding buffering for rate matching in the transport equipment complicates the design, increases cost, and increases system delay. Routers already utilize a significant amount of buffering for routing and rate matching between lower speed ports and high-speed trunk ports. Thus, routers are a logical place to handle the rate conversion between 10 Gigabit Ethernet LAN and WAN ports.

There are a number of possible scenarios for 10 Gigabit Ethernet in the POP. In most cases, the WAN PHY is employed between POP routers and transport equipment. However, the choice of PHY between routers and POP switches is hotly debated. Some ISP's will want the WAN PHY to interconnect routers and POP switches so that there is only one PHY type used in POP equipment. This reduces the chance for installation and maintenance mistakes. Other ISPs want to maximize the bandwidth between routers by using the LAN PHY.

Carriers are requesting dual-mode cards for POP equipment to reduce the risk of making the wrong PHY decision. A dual-mode PHY card allows a software change of POP equipment if necessary, rather than deploying field technicians to swap cards.

In addition to connecting routers, POP switches can also provide high-speed connections to Web hosting and data center servers for direct access to the Internet. This provides faster Web server response times by avoiding relatively slow access links that would be used if the Web servers were located at the business site.

SONET/SDH holds a huge installed base in metro architectures and will exist for a long time to come. Emerging metro architectures, such as optical switches/cross connects, are promising to reduce equipment costs and service provisioning time.

There are two main scenarios for emerging metro applications using 10 Gigabit Ethernet. Metro players with existing SONET/SDH equipment will most likely choose the WAN PHY as it allows management visibility from the network operations center using their existing SONET/SDH management tools.

New metro players, on the other hand, may choose the LAN PHY since the simple network management protocol (SNMP)is simpler to operate than SONET/ SDH management. Another debate rages over whether the LAN PHY with SNMP and basic fault management will be sufficient, or whether the more feature-rich SONET/SDH system and fault management is required.

The enterprise environment rounds out the 10-Gigabit Ethernet PHY applications. In enterprise applications, designers would most likely choose the LAN PHY due to its higher data rate and their own prior experience with Ethernet LAN style management. Information technology groups would typically not have SONET/ SDH testing tools, which would be a significant barrier to deploying WAN PHYs beyond a metro uplink card. The type of uplink would depend on the choice of the metro ISP architecture.

Staying ahead

To stay competitive, equipment developers must deploy scaleable network system architectures to allow a rapid and cost-effective response to increasing bandwidth requirements. Critical requirements in the deployment of Internet architectures include: support for emerging features, leveraging equipment improvements, and reducing management and service provisioning times.

The adoption of 10 Gigabit Ethernet, in addition to next-generation SONET/SDH and WDM equipment, will provide a significant step towards providing the infrastructure for high-bandwidth global data networks. The 10 Gigabit Ethernet WAN PHY allows frame and rate compatibility with existing OC-192 transport and optical infrastructure, whereas the LAN PHY optimizes the data rate for enterprise switching applications. The battle between the LAN and WAN PHYs continues, however, for the contested application areas, which include metro and Intra-POP connections.

Internet POP architectures are simplified by using POP switches, and direct connect to data center, and Web hosting servers via 10 Gigabit Ethernet switches. Ultimately, new metro/ access architectures based on 10 Gigabit Ethernet switching and optical cross-connecting applications will significantly increase metro bandwidth while reducing the provisioning times.

Stuart Robinson is a strategic manager of PMC-Sierra's 10 Gigabit optical networking products. Prior to his current position, Robinson worked for Telus, formerly BC Tel, in its advanced communications group. He graduated from the University of Victoria in 1994 with a degree in engineering and can be reached at stuart_robinson@pmcsierra.com.