This week we publish a new long-form report, Cloud e-discovery: litigation comes down to earth – download an executive summary here.

In cloud e-discovery we see two major market shifts: corporations in-sourcing e-discovery to lower costs, while outsourcing IT infrastructure and services around it through hosting.  Still in early adoption, it is a leap of faith on some level, and carries both risks and benefits.  While most users in our 2010 e-discovery survey were bringing the e-discovery process in-house, only 16% were using cloud to do it, for a variety of reasons including security, data loss, regulatory concerns, and ease of retrieval.

But consider that hosted e-discovery has actually been around for over 20 years. What’s more, while some enterprises are resisting the cloud, their law firms, service providers, and other outsourcers entrusted with their data are not.

Witness this month’s 2010 Am Law tech survey – 80% of law firms are using hosted technology, 60% of those for e-discovery.  In fact, e-discovery tops all hosted software usage, far surpassing HR (21%), spam filter/email (21%), storage (6%) or document management (5%).  And while 79% report a positive experience, 30% said the savings were not what they expected.  Limited customization, diminished data control and security were even greater concerns.

And what of the bigger-picture risks?  Cloud topped the agenda last month at the Masters Conference as well: the growth of public and private cloud data from mobile use and social media, potential regulatory pitfalls, the benefits and risks of hosted e-discovery, and growing cross-border issues.  No blue-sky thinking here, just hard truths on the cloud from those on the front lines.

From e-discovery lawyers and consultants:

• “[Public] cloud providers can’t meet the needs [of e-discovery] today.”
• “Your data, your problem.”
• “Data privacy in the EU is like free speech or freedom of religion in the US. . . they will give up the cloud before they give this up.”

From Microsoft General Counsel, speaking on cloud regulation:

• “Things will move quickly, and if something bad happens, things will move faster still.”

From an enterprise buyer on procurement:

• “It will take 19 months to work out e-discovery issues once you start talking about it.”
• “Every dollar they save on cloud will be three dollars in legal.”
• “I hate when people say ‘it’s not gonna stop – it’s already there.’ It makes customers think there is no choice but to comply.  But maybe ‘cloud’ will go away?”

And for the last word, a characteristically common-sense admonition from UK expert Chris Dale (speaking on ECA):

• “The best technology lies between your ears.”

So, how to navigate it all?  For a succinct analysis of the cloud e-discovery market, our report is available to 451 CloudScape or Information Management subscribers, or get an executive summary here.  It offers a market overview, benefits and risks of cloud e-discovery, adoption trends and inhibitors, market drivers, current vendor and service-provider offerings, and the future direction of the market, particularly for enterprise customers.

Also note a complementary report, Cloud archiving: a new model for enterprise data retention, by Simon Robinson and Kathleen Reidy.  They estimate the market will generate around $193m in revenues in 2010, growing at a CAGR of 36% to reach $664m by 2014.  This report covers growth drivers, the competitive landscape and the outlook for consolidation, featuring detailed vendor profiles and end-user case studies.

The cloud archiving market will generate around $193m in revenues in 2010, growing at a CAGR of 36% to reach $664m by 2014.

This is a key finding from a new 451 report published this week, which offers an in-depth analysis of the growing opportunity around how the cloud is being utilized to meet enterprise data retention requirements.

As well as sizing the market, the 50-page report – Cloud Archiving; A New Model for Enterprise Data Retention – details market evolution, adoption drivers and benefits, plus potential drawbacks and risks.

These issues are examined in more detail via five case studies offering real world experiences of organizations that have embraced the cloud for archiving purposes. The report also offers a comprehensive overview of the key players from a supplier perspective, with detailed profiles of cloud archive service providers, with discussion of related enabling technologies that will act as a catalyst for adoption, as well as expected future market developments.

Profiled suppliers include:

• Autonomy
• Dell
• Global Relay
• Google
• i365
• Iron Mountain
• LiveOffice
• Microsoft
• Mimecast
• Nirvanix
• Proofpoint
• SMARSH
• Sonian
• Zetta

Why a dedicated report on archiving in the cloud, you may ask? It’s a fair question, and one that we encountered internally, since archiving aging data is hardly the most dynamic-sounding application for the cloud.

However, we believe cloud archiving is an important market for a couple of reasons.  First, archiving is a relatively low-risk way of leveraging cloud economics for data storage and retention, and is less affected by the performance/latency limitation that have stymied enterprise adoption of other cloud-storage applications, such as online backup. For this reason, the market is already big enough in revenue terms to sustain a good number of suppliers; a broad spectrum that spans from Internet/IT giants to tiny, VC-backed startups. It is also set to experience continued healthy growth in the coming years as adoption extends from niche, highly regulated markets (such as financial services) to more mainstream organizations. This will pull additional suppliers – including some large players — into the market through a combination of organic development and acquisition.

Second, archiving is establishing itself as a crucial ‘gateway’ application for the cloud that could encourage organizations to embrace the cloud for other IT processes. Though it is still clearly early days, innovative suppliers are looking at ways in which data stored in an archive can be leveraged in other valuable ways.

All of these issues, and more, are examined in much more detail in the report, which is available to CloudScape subscribers here and Information Management subscribers here. An executive summary and table of contents (PDF) can be found here.

Finally, the report should act as an excellent primer for those interested in knowing more about how the cloud can be leveraged to help support ediscovery processes; this will be covered in much more detail in another report to be published soon by Katey Wood.

Our lengthy report that shares a title with this blog post hit the wire yesterday (a high-level exec overview is available here for all).  I’ve blogged before about our efforts on this.  It has been quite a project, with several months of listening, reading and talking with lots IT managers, attorneys, integrators, consultants and vendors.  Oh and writing — the final doc weighs in at 57 pages…

I noted before that I wasn’t sure “information governance” was a specific or real enough sector to warrant this kind of market analysis.  Aren’t we really just talking about archiving?  Or e-discovery?  Or ECM?  In the end, I found we’re talking about all these things, but what is different is that we’re talking about them all together. How do we ensure consistent retention policy across different stores?  How do we safely pursue more aggressive disposition?  How do we include all that “in-the-wild” content in centrally managed policies?

Is “information governance” really the right tag for this?  I don’t know, but I never came across anything better (I did toy with “information retention management” for awhile).  We  might be calling it something else in a couple of years, but the underlying issues are very real.

From the report intro:

What is information governance? There’s no single answer to that question. At a high level, information governance encompasses the policies and technologies meant to dictate and manage what corporate information is retained, where and for how long, and also how it is retained (e.g., protected, replicated and secured). Information governance spans retention, security and lifecycle management issues. For the purposes of this report, we’re focusing specifically on unstructured (or semi-structured,
like email) information and governance as it relates primarily to litigation readiness.

In the report, we look at why organizations are investigating more holistic information governance practices:

• to be better prepared for litigation
• to ensure compliance
• to reduce risks and costs of unmanaged or inconsistently managed information

Then we go into the market with analysis of:

• the rise of email (and broader) archiving for litigation readiness
• the relationship of the ECM and records management market
• Autonomy and other vendors advocating “in-place” approaches to governance

There are also sections on adoption issues, market consolidation and areas for technology innovation.  And profiles of 15 vendors (each with a SWOT analysis) active in this market.

Expect lots more on this topic moving forward.

As something of a follow-up to the special report we did last fall on the market for eDiscovery tools and technologies, we’ve begun work on a similar report meant to look more deeply at that first process phase in the EDRM — Information Management.

Information management sounds like a nice manageable topic, doesn’t it?

We’re looking specifically at the market for technologies meant to help organizations manage unstructured info (often ad-hoc, like email and unmanaged docs) more effectively so that eDiscovery won’t be such a firedrill if and when it occurs.

eDiscovery isn’t the only reason to get a better handle on this ad-hoc, unstructured info — there are compliance-related reasons in some cases and the costs and risks associated with storing lots of stuff for long periods of time when it should have been culled or deleted.  Conversely, not retaining information or at least having a documented retention and disposition plan is also risky.

As we’ve noted before, some are calling this “information governance.”  So is this a report on the information governance market?  Is there such a thing?

Here are some of the things we’re learning so far with our research:

• There’s no question that governance is a hot issue with many organizations.  Getting a better handle on email is the biggest pain point.  Check out this recent AIIM survey for some interesting data on this.
• Better preparedness for eDiscovery is the biggest driver, followed by the complexity of compliance, the need to reduce costs, and security concerns (security-related governance is really a separate market and not one we’re looking at here).
• One of the fundamental questions seems to come down to whether organizations want to take an archive-based approach to governance or one that is tied to an ECM platform.
• Since email is the big problem, email archives are a big part of the solution for many companies.
• Email archives are expanding to handle more diverse content types with more sophisticated retention, classification, legal holds and eDisco tools.
• The disconnect with this approach seems to be when emails or other content actually are records and need to be managed as such.  How data moves from one system (e.g., archive to records management system) or is managed in-place in an archive by an RM system seems to be mostly an unexplored issue for most organizatins at this point.
• Because of this, ECM vendors paint archive-only vendors as “point tools.”  ECM vendors see governance as an ECM problem and come at with platforms that generally include both archiving and records management.  But the archives from ECM vendors are generally newer or not traditionally as competitive in pure archiving scenarios.

All of the above makes for quite an interesting, if difficult to label, market.  We’re not really writing a report on the ECM market, since the archives are so critical to handling email especially, the major problem area, and most of the leading email archiving vendors are not full ECM vendors.  But there is definitely an ECM and records management component to this so we’re not just profiling the email archiving market.  In fact, we’re trying to only profile those vendors that can manage multiple content types and, ideally, do so across repositories.

Which I think leaves us talking about the information governance market.  This concerns me a little bit, as I worry that “information governance” is a vague tag and not really an identifiable sector.  But I see no other easy way to describe the intersection of vendors and technologies we see coming at this problem from different areas of strength.

I’d love any comments on what others think about this – is information governance a market?

Microsoft has begun to share information on what it calls the “waves” of Office 14 products set to hit the market this year and next. Most of the information at this point is on Microsoft Exchange 2010, which has entered public beta. General availability is expected in the second half of this year.

There’s also some info for SharePoint, though little detail. Microsoft SharePoint Server 2010 will go into technical preview in Q3 2009 and be generally available in the first half of 2010.  Beyond that, we still don’t know what will and won’t be in SharePoint.next (though we don’t have to call it that anymore).

The part of the Exchange 2010 announcement that caught my attention is the reference to an integrated e-mail archive.  Did Microsoft just enter the email archiving market?  That would certainly be noteworthy, given that much of the hot email archiving market involves archiving Exchange email.  Since Microsoft hasn’t had a horse in this race, this has been the realm of third-party providers like Symantec and Mimosa Systems to date.

On the analyst telebriefing held today by Microsoft on this announcement, I asked about this and the role for Microsoft’s email archiving partners going forward.  Michael Atalla, Group Product Manager for Exchange at Microft told me that Microsoft is out to meet the needs of the 80% of its customers that don’t yet have any email archiving technology and that existing email archiving products serve a “niche” of the market at the high end for customers that have to meet regulatory requirements for email archiving.

While I agree there is still a lot of opportunity in the email archiving space, describing existing adoption as limited to those in regulated industries isn’t exactly accurate.

I’ve tried to dig deeper into what this integrated archive includes.  Not easy, as there is no mention of archiving at all in the TechNet docs on Exchange 2010 (though there’s quite a bit of interesting detail on records and retention management).

Best I can tell, Exchange 2010 lets you create individual or “personal archives.”  This page from Microsoft explains that a personal archive is:

an additional mailbox associated with a user’s primary mailbox.  It appears alongside the primary mailbox folders in Outlook. In this way, the user has direct access to e-mail within the archive just as they would their primary mailbox. Users can drag and drop PST files into the Personal Archive, for easier online access – and more efficient discovery by the organization. Mail items from the primary archive can also be offloaded to the Personal Archive automatically, using Retention Polices…

So it moves the PST file from the desktop to the server, which makes it more available for online searching and discovery purposes.  But is that really email archiving?  I can see how that would be attractive to end users that want an easier way to access archived emails, but it seems like it would increase the load on the mail server and not handle things like de-duping, which archiving is generally meant to address.

I’m not an expert on email archiving though.  I’d love to hear from anyone who has comments.