The InfiniBand Architecture for 2003 and 2004

In August of 1999 all of the major server vendors combined to develop a new I/O "fabric" for servers and storage systems within the enterprise data center called the InfiniBand Architecture. This fabric's goal was to provide IT managers with a better way to deploy, scale, and manage computing power within the data center.

Today, the InfiniBand architecture is meeting this goal and is the only 10Gb/sec transport that enables industry standard servers to be clustered together for reliable, available, scaleable and high performance enterprise computing. InfiniBand and Linux are the perfect marriage for distributed, industry standard computing. InfiniBand connected system-to-system fabrics deliver lower capital and operating costs by enabling the following:

• Industry standard servers and storage systems can be clustered together without the cost or proprietary lock-in of vendor specific solutions.
• InfiniBand accomplishes this at 10Gb/sec with ultra low clustering RDMA latencies that enables break through performance for both enterprise data base applications and High Performance Computing (HPC).
• InfiniBand products, featuring 2nd generation silicon, are available today from many suppliers.

Figure 1:  InfiniBand and Gigabit Ethernet compared. Source: Oracle

HPCC: The High Performance Compute Clustering (HPCC) market is a vital IT requirement for many companies. These clusters are used for such applications as: complex financial analysis, automotive design and crash tests, seismic simulations in gas and oil research, bioscience or molecular modeling, industrial simulations, CAD, graphics rendering, and others compute intensive tasks. HPCC might not seem like a standard Enterprise Data Center (EDC) application but many corporate data centers feature these applications as they represent the core R&D function within many companies.

In addition, HPCC is a great bellwether for Enterprise Data Center performance. Initial benchmarks available have demonstrated 8X the bandwidth vs Ethernet delivered with an order of magnitude lower latencies. MPI over InfiniBand has shown better latencies (sub 7 µsec) and greater bandwidth than that of the proprietary HPC clustering technologies (Figures 2 and 3).

Figure 2:  MPI bandwidth vs. message size. Source: Ohio State University, Department of Computer and Information Science

Figure 3:  MPI latency vs. message size. Source: Ohio State University, Department of Computer and Information Science

	To provide the utmost flexibility Mellanox has designed a modular HCA that enables rapid local bus adaptation to match future server chip sets, whether it be PCI-X 2.0, PCI Express, or any other local interface. Today, Mellanox InfiniHost HCAs feature PCI-X with PCI-Express and PCI-X 2.0 under development as the server interconnect options over the next year (for more information, see Mellanox's IDF Spring 2003 presentations). This flexibility of design will enable InfiniBand connections to all the major server chip sets scheduled for 2003 and 2004.

• Time-to-Market:
  InfiniBand technology is a generation ahead. Second generation 10Gb/s RDMA InfiniBand products are in the market today. By contrast, the first Ethernet RDMA specification won't be completed till fall 2003, with first generation products due in 2004. By that time 12X or 30 Gb/sec third generation InfiniBand products, again with RDMA, will already be deployed in the market.
  
  
• Cost:
  Multiple vendors have introduced, interoperable10Gb/sec InfiniBand switches with port costs from $600 to ~$1000 per port. These 10Gb/sec InfiniBand products can currently be purchased. InfiniBand dual port 10Gb/sec host channel adapters are already in the market at $999. By contrast Ethernet RDMA products are due till 2004, and the current 10 Gb/sec Ethernet NICs are $7,995 each with switch ports starting at $12K.
  
  
• Complexity:
  InfiniBand was designed specifically to support low latency RDMA semantics for clustering. Other technologies, such as Ethernet, must overcome legacy issues to support this critical technology. This is a key reason why InfiniBand will always cost less than 10 Gb/sec Ethernet with RDMA and a TCP offload engine.

• Enterprise Data Center (EDC): Major enterprise data bases, like Oracle and DB2, support many of the current enterprise IT advancements and will support InfiniBand. Also, this market includes InfiniBand as an interconnect technology for server blades.
  
  
• Native attached InfiniBand storage: NetApp and Fujitsu are leading a high performance easy to use file based InfiniBand storage effort.
  
  
• High Performance Computing Clustering (HPCC): Utilizing a number of industry standard servers for HPC clustering for compute intensive corporate R&D or academic research efforts.
  
  
• Embedded Applications such as the long standing VME specification, PICMG standard, and other applications like video streaming, military, storage fail-over, Advanced TCA for Telco and others.

1. HPC clusters will be deployed in mid 2003. These clusters are straightforward to deploy and the software needed is ready. There are multiple versions available of an InfiniBand aware protocol called Message Passing Interface (MPI) that allows existing HPPC applications to take full benefit of InfiniBand performance without modification.
   
   
2. Deployment of clusters of industry standard Linux servers that feature huge amounts of raw processor and memory performance running enterprise databases that are deployed with lower capital expenditures and enable lower operations costs. These clusters can be thought of as: "how to get ALL your database into memory without a lot of expense." These clusters will soon be complemented by native InfiniBand DAFS storage and accompanied by existing I/O bridges for communication within the rest of the data center. Offering clustered database solutions at a fraction of the cost of mid-range and high-end server solutions is the key for InfiniBand achieving a solid beachhead in the data center. This begins to take hold in late 2003 and early 2004.
   
   
3. Major server OEMs replace their existing internal server clustering technologies with InfiniBand technology. In the past, the major OEMs have made huge engineering and monetary investments to develop, advance and deploy scalability and reliability capabilities within their own proprietary mid-range and high-end server systems. Expect these transitions in 2004.
   
   
4. As the Data Base and mid tier applications dependencies grow to utilize InfiniBand and expand the volume of the market, third generation products will drive down the cost of InfiniBand and expand the infrastructure. This maturity will prove the event driven self-healing capabilities of InfiniBand and expand the use of the technology in the Enterprise Data Center from the data base tier, to the applications tier and finally to interconnect with the edge of the data center.

Concurrent to the four major phases, embedded designs have been deployed into production today by companies including Network Appliance, Sky Computers, SBS Communications, Themis Computer, Ramix, and VMIX. Many others will be deployed in 2003 and 2004.

The current 10GE switch products are $12K (or more) for each port! But, this may be somewhat misleading as 10Gb/sec Ethernet is not a data center technology but instead designed for long haul WAN connections. But many still contend that this technology will migrate from the WAN/MAN to much higher volume server and desktop computers. But due to the high cost, complexity and the lack of backwards compatibility, 10GE technology is not expected to be in personal computers any time soon, if ever. Thus the cost curve is not likely to follow the high volume path that 100Mb or 1GB Ethernet has enjoyed in the past. Adding RDMA and TCP offload engines will makes 10GE even more complex and costly.

Again, InfiniBand includes hardware transport, kernel bypass and RDMA that all results in low processor utilizations for InfiniBand. When running data base applications that charge licensing fees by processor, the IT manager wants every possible CPU cycle used on the application, and not on the transport.

In the near future the combination of InfiniBand, Linux, industry standard servers and clustered enterprise data bases will make a huge impact on how servers are purchased and deployed, how they perform, as well as operate within enterprise data centers.