Stephen Lawson (IDG News Service) 11 August, 2009 06:53:00

Cloud computing and flash-based storage, two of the fastest-growing technologies in IT, may drive each other forward as Internet-based service providers demand faster access to large amounts of data.

Flash storage has lower reading latency than hard disk drives because it doesn't need to spin a disk to get to a particular bit of data. With SSDs (solid-state disks) and PCI Express flash cards, it's possible to read data anywhere in the device in less than a millisecond, compared with several milliseconds on a hard drive.

This can have some utility in individual enterprises, but the real benefit comes at the scale of public cloud computing, where one service provider may be delivering data to hundreds or thousands of customers at the same time.

Though flash still has some hurdles to overcome on the road to wide deployment, some Web-based services companies see it as the most promising advance happening in IT infrastructure.

"In the last 20 years, spinning disk really hasn't gone any faster, and right now we're really on the cusp of a change with flash technologies," said Richard Buckingham, vice president of technical operations at MySpace, at the Structure conference in June in San Francisco.

MySpace rival Facebook praised flash at the same conference.

"Flash is going to have a very, very significant effect on not just storage, but infrastructure as a whole. And I think it's going to have at least as significant an impact as going from single-core to multicore CPUs," said Facebook Vice President of Technical Operations Jonathan Heiliger.

Internet-based companies expect to use flash in different ways, but scale is a major concern for all of them. It's this factor that could turn flash into the game-changer at online companies that it won't be for most enterprises, according to Forrester Research analyst Andrew Reichman.

SSDs and flash storage cards take up less space and power than spinning disks, even while they deliver bits faster. Cloud-based companies' data centers are so big that these benefits really matter, he said.

"If it was just a single company, the likelihood is that the performance would matter less," Reichman said.

Solid-state storage has long been a part of data centers that need low latency. Since the 1990s, financial firms and other companies have stored large amounts of transaction data in DRAM for quick access, said storage consultant Tom Coughlin.

Flash isn't quite as fast as DRAM, but it's less expensive, uses less power and holds onto its contents whether it's powered or not. As a result, IT managers are starting to see it as a more affordable path to fast reads of certain types of information, such as metadata, transaction data and bits needed for transactions.

Even online entertainment companies are starting to get interested in flash storage for their content, Coughlin said. Flash is available in both SSDs, which come in the same form factor as hard drives, and smaller flash cards from vendors such as Fusion-io, which can be plugged directly into servers using PCI Express interfaces.

Most enterprise flash is made with a single-level architecture, which is more expensive and not quite as dense as the multilevel flash used in consumer devices such as iPods.

Multilevel flash uses different levels of voltage for different bits of data, which allows more densely packed data but requires extra management that reduces real-world performance, Coughlin said. Many observers consider single-level more reliable and less prone to losing capacity over long periods of heavy use.

Major storage vendors such as EMC have been offering flash options since last year. Most get the actual drives from STEC, a solid-state storage manufacturer in Santa Ana, California, and integrate them into their own systems.

In April, EMC introduced Symmetrix V-Max, its first storage platform designed from the start to use SSDs as well as spinning disks. Symmetrix is particularly geared toward high performance, but SSDs are also available on the company's other storage systems.

IBM offers STEC SSDs up to 146GB in its DS8000 Enterprise Disk Array. It also plans to let users integrate SSDs into its SAN Volume Controller appliance, which can manage many different types of storage.

Flash should be available for all of IBM's enterprise storage platforms by the end of this year, said Clodoaldo Barrera, chief technology strategist for IBM System Storage.

Hewlett-Packard offers SSDs for its high-end XP storage arrays and midrange Enterprise Virtual Array (EVA), as well as flash cards made by Fusion-io that fit into HP servers. In cases where performance is key, the cost advantage can be significant, according to Kyle Fitze, director of marketing for HP Storage Platforms.

Enterprises commonly gang together several lightly loaded, high-speed Fibre Channel hard drives to take advantage of their combined speed, a process called short-stroking. On the EVA, eight 72GB SSDs can outperform 324 Fibre Channel drives, each 300GB with 15,000 RPM speed, Fitze said. What's more, the whole package would typically cost $US605,000 with hard drives and $US155,000 with SSDs, he said.

SSDs still cost 25 times as much as spinning disks on a per-gigabyte basis, according to IDC analyst Jeff Janukowicz. But the need for performance, capacity utilization and lower power consumption, especially by large-scale cloud operators, should drive enterprise SSD sales up an average of 165 percent per year until 2013, he said.

Gartner sees a similar explosion, with sales of 59,000 units last year growing to 281,000 this year.

Some of the biggest cloud service providers may help to drive that growth.

Along with fast access to key pieces of data, Facebook anticipates "tremendous" gains in both reliability and lower power consumption from flash, operations chief Heiliger said. The social-networking company, with more than 200 million users, is highly demanding in its data centers.

At the Structure conference, Heiliger criticized server manufacturers for not delivering systems designed from the ground up for power savings, and Intel and AMD for not meeting performance promises on some new chips.

MySpace is interested in using flash to save data-center space while maintaining fast page loads for its users, according to Buckingham. One way MySpace is likely to do that is by replacing short-stroked disks with flash.

That technique should enable the company to use a server that's one rack unit (1U) high instead of 2U, which could make a big difference in a company with 60,000 square feet (5,600 square meters) of data-center space, he said. The company has petabytes of storage, and unlike most enterprises, it has to respond to hundreds of thousands of page requests per second, he said.

MySpace would use flash as a cache for frequently used data in its databases, as well as for maintaining indexes for search, Buckingham said. The company has tested both single-layer and multilayer products and is attracted to multilayer because it costs less.

But MySpace wants to make sure the storage doesn't degrade under the heavy workloads it would be subjected to.

Buckingham has been working with several vendors to establish baselines for performance and reliability. Even so, MySpace won't rely on flash for persistent data, such as the actual pictures that users post on their pages. Only about one-twentieth of the company's data would ever be stored on flash, Buckingham said.

"I'm never going to write something to an SSD and hope it lasts forever," Buckingham said.

NetSuite, a provider of on-demand software for uses such as enterprise resource planning, customer relationship management and e-commerce, sees flash as a place to store heavily used database information for its customers.

That data helps to drive "real-time dashboards" that show NetSuite's medium-sized enterprise customers current information such as their expenses, sales forecasts or number of open customer service tickets, said David Lipscomb, senior vice president of engineering.

NetSuite wants to let users refresh the dashboard as frequently as they need it, which can demand a lot of speed if thousands of customers are using their dashboards at the same time. Flash should be ideal, he said.

"When this thing comes along that allows you to make the database so much faster, you say, 'That's something I need to drill on,'" Lipscomb said.

NetSuite is experimenting with flash cards from Fusion-io and finding they offer between two and five times the performance of hard drives, Lipscomb said. SSDs are still too expensive for the company's data-center architecture, which is built around large numbers of standard, inexpensive components, he said.

Despite its benefits, flash storage is expected to remain just one tier of a total storage infrastructure at both traditional and Internet-based enterprises. Aside from questions about long-term reliability, one of the biggest remaining challenges for the technology is how to put the right data in it.

Ideally, what resides in flash should be the most urgently needed data at any given time. Vendors are working on software to automatically allocate data to flash, fast hard drives and slower drives.

EMC plans to introduce its Fully Automated Storage Tiering software by the end of this year. HP is also working on automated tiering.

Cloud computing companies are likely to be at the cutting edge of overcoming those challenges as they try to keep up with booming Internet activity.

"The decisions they make, or the innovations they create, are likely to change the market in a big way," Forrester's Reichman said.

As CIO for Sun Microsystems, I'm of course expected to manage all of the traditional aspects of IT: hardware, software, applications, employee productivity, future technology investments, etc. But given the significant R&D and engineering investment my company has made to design and create energy-efficient technology, I've had the advantage of being an early adopter of such technology and have installed several highly energy-efficient data centers recently. Here are a few things I have learned in the process:

Use ecofriendly technologies whenever possible when upgrading your server and data center hardware. Even if your department does not directly bear the brunt of high electricity or facilities fees, be sure to calculate the full cost of ownership, including space, power and cooling costs into your purchases, and take full advantage of any applicable utility company rebates. I speak from experience when I say that these alone saved my company nearly a million dollars during recent upgrades.

Next, if you are designing new facilities, take advantage of the opportunity to realistically evaluate your space needs. Today's energy-efficient technology takes up significantly less floor space than previous generations of hardware. Virtualization makes location nearly irrelevant.

Work together. Understand the financial impact of technology on operating expenses and create a plan together. Talk with your facilities organization to identify consolidation opportunities. The technology is readily available, so by incorporating best practices in data center design, you can both save your company money and reduce its environmental impact. Make sure those who run IT and who manage space have standard efficiency metrics to arrive at a complete cost picture.

New standards in data center design are not only good for the environment, but they are also good for the bottom line. By working together, CIOs and CFOs can direct their efforts to successfully implement green solutions.

Earlier this month the U.S. Environmental Protection Agency issued a report to Congress that was intended to send a message to U.S. businesses: run your data centers more efficiently. No doubt, the EPA is recalling the success of its consumer-oriented Energy Star Program in this assessment of data center power consumption and suggestions for improvements. Given the EPA's findings that server and data center power consumption in 2006 is estimated to be more than double the electricity that was consumed for this purpose in 2000 (PDF) and shows no signs of slowing, it's surprising that there has not been more heated debate in the boardroom between data center managers and chief information officers and facilities managers and chief financial officers.

Large corporations consistently shoulder substantial electricity and real estate costs. As managers, we should expect to receive more corporate scrutiny as the resources we consume for massive technology deployments skyrocket; that is, if we have not already begun to feel the heat, so to speak.

In addition, utilities in some parts of the country such as California and New York have already stretched their resources thin to satisfy the growing power needs of local businesses. When you consider Gartner analyst Michael Bell's prediction that by 2008, at least 50 percent of data centers will have insufficient power and cooling capacity, ("2006 Data Center Polling Results: Power and Cooling") it's not hard to see the imminent clash.

IBM plans to shift investment, $1B annually, to make its clients and itself more energy efficient through achievement of green data centers. This will result in new products and services in the information technology domain. IBM expects to deliver assessments, create better data centers that leverage virtualization technology and advanced cooling, and manage power in an optimal way. ... ... "The savings are substantial—for an average 25,000 square foot data center, clients should be able to achieve 42 percent energy savings. Based on the energy mix in the US, this savings equates to 7,439 tons of carbon emissions saved per year. Called Project Big Green, IBM’s initiative targets corporate data centers where energy constraints and costs can limit their ability to grow. The initiative includes a new global green team of more than 850 energy efficiency architects from across IBM. " ... Via IBM: Green Data Center