The Coming Data Deluge

The top vendors in the overall storage software market are led by EMC with 24% share in 2008, followed by Symantec at 18%, IBM at 13%, and NetApp at 8%.  Interestingly, although these players command substantial presence smaller companies continue to chip away at market share.  I would not be surprised if, in ten years, the list of top ten vendors shuffles around quite a bit.

According to IDC, the total data created by businesses and consumers is roughly doubling every 18 months.  At this current rate, the universe of content created across the globe will grow five-fold by the end of 2012.  And, there’s no clear end in sight. It doesn’t take much to understand this when one starts to think about their own personal storage growth over just a few months.  Among others, the obvious following factors are playing a key role in this data deluge:
  • Global growth in Internet users, broadband penetration, and Internet-connected devices (both in terms of multiple computers per user and, now, powerful mobile devices).
  • Adoption of social networking applications and the exponential growth in media (photos, video, & music) that gets duplicated & distributed across the network.
  • Increasing bandwidth availability (which enables richer and bigger applications, especially video).
  • Migration of content stored on old media (paper, film, etc.) into digital format (movies, music, legal files, letters, books, magazines, etc) and the coming digitization of healthcare and other records.
This increasing deluge of digital data, I believe, is bolstering a continuously growing demand for focused storage software (and services), starting with traditional backup & recovery (with improved functionality across platforms & media), replication, archiving, and storage management.

Another important data point to consider is that today corporate organizations spend, on average, 28% of IT capital expenditures on data storage hardware, software, and services.  That’s a massive percent.  Given that a large majority of that is hardware today, how will that percent evolve over the next ten years as data continues to grow exponentially and historic data continues to accumulate?

Put into perspective from a consumer angle; think about what your files would look like if you had kept every single photo you ever took, every letter/paper you ever wrote, every card you ever received, every drawing you ever drew, and every document you ever saw or read.  Had you kept all of that material/ media for your lifetime, how would you manage it today?  How would you go about finding something?  How would you decide where to file something?

In this digital age, where we are individually clearly keeping more personal media than we ever have before (just think about the number of digital photos you have on your hard drive versus three years ago),  we are individually spending much less, as a percent, than corporations do.  Our spending on media & communications has dramatically increased over the past twenty years, but our personal spending on filing & storage of that media has not increased in a corresponding manner.

This material increase in data clearly presents a huge opportunity for continued innovation in capacity optimization technologies and more efficient system architectures to address this massive piece of our IT ecosystem (both in corporate environments and in the consumers’ living room).

Storage is a key component of this wave of digital media innovation we are living in.  So, if you ask me, the innovative companies in storage software are going to appear & grow mainly because of:
  • As mentioned above, absolute growth in bits & bytes, highlighting increased for storage management and protection processes, both for ease of access and for cost reduction.
  • Cloud computing, which is shifting the physical location of data, and demanding new tools & solutions to deliver storage-as-a-service, archiving & de-duplication, and multi-tenant management.
  • Increasing platform & application complexity, where we are moving from a singular platform (Microsoft) and a handful of application standards (text, images, email) to multiple platforms and applications.
This last point is critical.  A few years ago, we lived a world where we basically backed up Microsoft and, mainly, MS Office applications (along with some basic 2D & 3D media files).  Today, storage software has to deal with heterogeneous environments (for instance, various kinds of mobile data devices, networked computers, netbooks, and Apple, gaining ground in the desktop world), and an increasing number of file types & media.

There is clearly a lot of innovation yet to come in an industry that many view as mature.

Kurzweil's Transcendent Man

A few years ago, at some technology conference, I received & read The Singularity is Near by Ray Kurzwell, and all I could think of was Skynet from the film series The Terminator.   The book insightfully explores the merging of technology and biology, and its implications for the future. The 672 page tome dives into nerdy augury of the coalescence of genetics, nanotechnology and robotics.  As a Blade Runner fan, Kurzweil's predictions of evolution are a scientists wet dream.



Paul Kedrosky recently posted the above trailer and it's gotten me excited to to see the documentary, Transcendent Man.  As in the book, Kurzweil argues that artificial intelligence is improving exponentially, and eventually – the latest ETA is apparently 2045 – computing will become self-conscious and "alive".  Bring it.

Mobile Operating Systems

Yesterday, Google introduced its long-awaited touch-screen phone, called the Nexus One.  This long-awaited new device/announcement is driving some chatter about the future of the Mobile OS world.  Much of the chatter is around pricing (to carriers), revenue model, and market share.  Google is effectively paying carriers to sell it's phone (platform) with the intent to generate revenues from search down the road.  That's a substantially different approach than Apple took with the iPhone.

Rather than re-hashing all the ins & outs of Apple versus Google, Bill Gurley does a good job here.

But, the question to ponder is whether the market will evolve as Macintosh and Windows did or whether Apple will hold ground & continue to grow. Will, over time, the Mobile OS world sustain multiple providers or will we end up in a world where one OS dominates market share?

Is there a developer, application, and user network effect in the mobile world? Any kind of network effect could impact this outcome uniquely.

On the face of things, it would seem to me that Google is taking a page from the Microsoft Windows playbook and Apple is taking a page from, well, the, uh, not so successful Apple Macintosh playbook.

It could also just be that the Mobile OS world can end up looking very much like the video game console industry, with three players (Microsoft XBox, Sony PlayStation, & Nintendo Wii) today equally vying for market share along with several other players (Sega, Atari, 3DO, NEC, RCA, etc) making waves here & there.

What about RIM, Nokia, Motorola, and Symbian in this above discussion? And, dare I say, Palm?  Is it sound to simply assume that these players are out of the running?

Although more free market examples would imply a standard Google & Microsoft analogy, only time will tell how the Mobile OS world evolves differently.  As mobile applications become more complex, 3G connectivity becomes more pervasive, and devices become more powerful, the Mobile OS landscape will also become more expensive to compete in.  For now, though, my guess is that it's still pretty early to start predicting the next Windows.