The demand for storage remains robust, though disk drive makers are often hard pressed to reap significant revenue and profits from it.
The amount of new stored information grew 30 percent between 1999 and 2002, according to the latest study from the School of Information Management and Systems at the University of California, Berkeley. Ninety percent of that new data was stored on hard disk drives, according to the study.
The study estimated 1,693,000 terabytes of information are produced and stored magnetically worldwide each year. That includes an estimated 400,000 terabytes of e-mails and 5,000 terabytes of data shared on KaZaa, a peer-to-peer site popular for pirated music.
Hard drive makers are more than keeping up. Hal Varian, a professor at the Berkeley school, noted that the growth in new information is generally at a slower rate than the growth in capacity of hard disk drives.
Indeed, not only is drive capacity more than keeping pace with data growth, but drive prices also often fall faster than demand rises. During the downturn, hard disk makers saw sales fall 18 and 9 percent and average selling prices drop 15 and 18 percent in 2000 and 2001, respectively. Market watcher iSuppli (El Segundo, Calif.) estimated drive makers returned to revenue growth in 2003 and will see a modest compound growth of just 3.5 percent through 2007, factoring in rising demand and still-falling prices.
Researchers are fueling plenty of sources to keep new data flowing, based on presentations at a recent symposium in San Jose sponsored by IBM Research.
Gordon Bell, the designer of Digital Equipment Vax and now a researcher at Microsoft Corp., described his efforts to commit all his personal information to digital storage. His terabyte/person target will need to grow now that he is including videos and phone calls as well as data, e-mails and pictures, Bell said. Head-mounted cameras and GPS units are next (see research.microsoft.com/CARPE2004). One objective of the research is to discover what kinds of search, linking and annotation tools Microsoft might want to build for a world in which people save all their personal data digitally.
"You don't keep everything because you want to look at it all, you keep it because you can never predict what part you will want to look at," said Microsoft researcher Jim Gemmell, who works with Bell.
At the same event, consultant Andreas Weigand talked about how consumer Web sites such as Amazon.com, where he was once chief scientist, conduct extensive analysis of the roughly terabyte of data a day they collect from people using their site. Weigand said that the analysis is part of a strategic new field of "computational marketing."
Going a step further, Web pioneer Brewster Kahle described his efforts trying to save for posterity all available information on the Internet. His Internet Archive project in San Francisco (see www.archive.org) collects about a petabyte a year and is doubling its storage almost annually.