How to build serial ATA storage arrays

The emergence of serial ATA (SATA) offers the promise of very large storage arrays with a price/performance ideal for near-line storage of business information, such as archived emails, corporate reference materials and other information that does not require frequent access. But such systems are sufficiently different from arrays based on SCSI and Fibre Channel that they present their own set of design challenges particularly in signal integrity, enclosure management, hot plugging and power distribution.

SATA arrays offer higher-capacity and lower cost than arrays based on parallel ATA. And, with data rates up to 1.5Gbits/second in its first version, their performance is well suited for the low- to mid-range application space.

The Serial ATA II extensions to the SATA 1.0 standard were released in October 2002 to provide the architectural skeleton for command queuing, the ability to monitor the enclosure environment, and define electrical design guidelines for backplane operation. Large numbers of drives can now be addressed by following the SATA Port Multiplier specification released in February 2003. With these specification extensions, the pieces are now in place to create SATA storage arrays.

These arrays, however, have unique signal integrity issues. Designers of storage arrays only see the possibility of high frequencies of serial transmission on a large number of parallel traces with SATA. That's because Fibre Channel uses a different routing scheme than SATA, and SCSI signals travel at much slower rates than SATA.

The maximum capacity for a SATA storage array would be a 15-drive rack mount unit with a single host connection at one end. In this case, we would have 30 differential pairs running in parallel, ranging in length from less than two inches to the first drive slot to a maximum of 18 inches to the last drive slot.

Even with no active host commands, all links are transmitting at a base frequency of 750MHz. Think of 30 small AM radio antennas within a three-inch span, all transmitting different programs on the same frequency, yet not allowed to interfere with each other. Of course things aren't really that bad — differential cancellation takes care of most of the interference — but that multiple radio station image can help stimulate some creative solutions.

The most direct way to mitigate the problem is to change the number and placement of fan-out points. Shorter traces reduce the opportunity for mutual interference and simplify the physical layout of the array. Designers can reduce the number of parallel traces and the maximum trace length by 50 percent and 75 percent, respectively.

Ultimately, the degree of signal integrity achieved in a backplane design can be stated as a single number, Bit Error Rate (BER). This is a functional measurement. The electrical measurement that can be directly correlated to BER is jitter in the serial bit stream. The minimum allowable bit error rate for SATA is one error in 1012 bits. This is usually spoken of as a 10 -12 BER. The jitter specifications to achieve this are documented in detail in the serial ATA standard (www.serialata.org).

Mandatory monitoring

Testing a backplane for compliance with electrical requirements should be an essential part of product design. Development kits and reference designs from chip vendors can help. Collaborative testing with disk drive vendors is another avenue to explore.

Almost all storage arrays today incorporate hardware to monitor the operating environment and control LED or LCD indicators and displays. Many storage array designs control power sequencing to the disk drives as well. The most common software interface to such enclosure management hardware is defined by an industry standard called SAF-TE (SCSI Accessed Fault-Tolerant Enclosures). A more comprehensive interface is defined in the SCSI standards document, SCSI Enclosure Services. The SATA II Extensions support the same management functions that are available on SCSI or Fibre Channel arrays.

For SATA storage arrays, one must also decide what monitoring and control services to provide within the cost constraints of a proposed product. Internal temperature monitoring, possibly from sensors at several different points could be essential. Voltage levels are likewise universally monitored. Audible or visual alarms indicate out-of-range conditions.

Designers can support other features including fan control, individual slot power control, hot plug detection, staggered spin-up and activity and error LEDs.

Hot plugging

Unlike parallel ATA, the SATA power connector was designed to permit hot plugging, with three levels of contact mating during connector insertion. The first mate level is a ground contact, the second is a pre-charge contact for each of the three voltages (12V, 5V, and 3.3V), and the third level is the remaining ground and voltage contacts.

The primary backplane electrical consideration for hot plugging is the possible need to limit in-rush current on the pre-charge contacts, especially 12V. Guidelines for specifying a pre-charge current limiting resistor value are given in the Serial ATA Extensions document, which can be found at www.serialata.org/ along with the base Serial ATA specification.

Another consideration for supporting hot plugging is detecting the presence of a newly inserted device. A fully compliant SATA II storage array can use hardware or firmware to set a newly defined bit in SError, one of the host accessible SATA registers. The bit indicates that the installed state of the related slot has changed.

The SATA power receptacle has contacts assigned to 12V, 5V, and 3.3V. It was intended to be a single design serving notebook computers, desktop computers and storage arrays. Each of these voltages is used by some classes of disk drives but no drive manufactured today uses all three. Almost universally, the 3.5-inch disk drives used in desktops and storage arrays today require 12V and 5V.

While initial SATA drives will undoubtedly continue the 12V/5V tradition, we will see drives in the future that use all three voltages. Having 3.3V available directly will simplify layout and reduce component count for the drive manufacturer. It will also save power consumption by eliminating 5V to 3.3V conversion on the drive.

However, the array designer's job is a bit more challenging if the designer wishes to deliver all three voltages in addition to ground to the drive slots. A simple solution is to add another power plane layer to the backplane, but this adds cost to the backplane.

An alternative is to distribute three voltages instead of two on the existing power plane. 3.3V is more difficult to distribute with stability than 5V or 12V, and it is always preferable to distribute power at higher voltages to minimize Ohmic losses. It takes 50 percent more current at 3.3V than at 5V to deliver the same amount of power. This translates into bigger variations in the delivered supply voltage due to IR losses in distribution. Adding a conductor for 3.3V distribution takes metal away from the distribution for 12V or 5V, with consequent higher resistance on those paths.

The operating characteristics of disk drives present a very uneven load demand profile to the power supply. Designing a power distribution system that can accommodate such variations and deliver power at three supply voltages will be a challenge on storage arrays with a large number of drives. The question for the array designer is whether to design a simpler two-voltage power system for today's drives or a more complicated three-voltage power system that anticipates requirements of later generation drives. The best answer will combine your own product goals with the product roadmaps of your disk drive supplier.

See related chart