I attended the inaugural Enterprise Search Europe in London October 24-25 and was impressed with a few things. Firstly the attendance was stronger than I thought, perhaps around 100, although that includes sponsors’ attendees. There were half a dozen main sponsors and two vendors I spoke to said they wanted to sponsor it but were too late – so there is demand there for such an event. Chairman Martin White of Intranet Focus did an excellent job of keeping things moving along and stepped in to fill a gap in the program caused by last-minute absentees. He’s summed up the whole conference himself here.

I was on two panels, one more formal than the other. I was responding to the thoughts of Funnelback’s chief scientist David Hawking, who talked of current research in the enterprise search area and contrasted it with the wealth of research done in Web search. I pointed out that if some organizations such as TREC made its output actually usable by vendors (who aren’t permitted to say where they come in TREC rankings) then it would get much more support.

In the evening in the Hand & Flower pub across the road from the hotel for the latest Enterprise Search Meetup London I was one of two permanent panelists (Laura Wilber of Exalead being the other), with a goldfish bowl set-up where anyone wanting to ask a question had to joint the panel’s spare seats to do it from there. It made for some pretty lively discussions about what the next big things might be impacting enterprise search (big data being one of them, of course), helped by the  beer & wine paid for by Exalead.

The evening in the pub might have been livelier than the panel at the conference itself, but the themes were similar, namely that enterprise search as a use case and value proposition is dead; people don’t buy enterprise search engines to ‘find stuff’ within their organization anymore. There has to be a more tangible use case, such as electronic discovery, or fraud detection and the like. I’ve written about this recently.

Two other analysts were at the conference – Alan Pelz-Sharpe of RealStoryGroup and Mike Davis of Ovum. Alan has penned his forthright thoughts here and Mike and I talked enough to know that the three of us – and plenty of others at the event that weren’t analysts but actual users – were thinking similar thoughts. So despite the fact that the conference is called Enterprise Search Europe and most people seem to think that enterprise search as a concept is redundant, I would recommend the event for next year, by when I suspect it may be called something else.

Customers of The 451 Group would have seen my report on the enterprise search market published September 15. If you are a client, you can view it here. I thought it would be useful to provide a condensed version of the report to a wider audience as I think the market is at an important point it in its development and it merits a broader discussion.

The enterprise search market is morphing before our eyes into something new. Portions of it are disappearing, and others are moving into adjacent markets, but a core part of it will remain intact. A few key factors have caused this, we think. Some are historical, by which we mean they had their largest effect in the past, but the ongoing effect is still being felt, whereas the contemporary factors are the ones that we think are having their largest impact now, and will continue to do so in the short-term future (12-18 months).

Historical factors

• Over-promising and under-delivery of intranet search between the last two US recessions, roughly between 2002 and 2007, resulting in a lot of failed projects.
• A lack of market awareness and understanding of the value and risk inherent in unstructured data.
• The entrance of Google into the market in 2002.
• The lack of vision by certain closely related players in enterprise content management (ECM) and business intelligence (BI).

Contemporary factors

• The lack of a clear value proposition for enterprise search.
• The rise of open source, in particular Apache Lucene/Solr.
• The emergence of big data, or total data.
• The social media explosion.
• The rapid spread of SharePoint.
• The acquisitive growth of Autonomy Corp.
• Acquisition of fast-growing players by major software vendors, notably Dassault Systemes, Hewlett-Packard and Microsoft.

The result of all this has been a split into roughly four markets, which we refer to as low-end, midmarket, OEM and high-end search-based applications.

Entry-level search

The low-end, or entry-level, enterprise search market has become, if not commodified, then pretty close to it. It is dominated by Google and open source. Other commercial vendors that once played in it have mostly left the market.

The result is that potential entry-level enterprise search customers are left with a dichotomy of choices: Google’s yellow search appliances that have two-year-term licenses and somewhat limited configurability (but are truly plug-and-play options) on the one hand, and open source on the other. It is a closed versus a very open box, and they have different and equally enthusiastic customer bases. Google is a very popular department-level choice, often purchased by line-of-business knowledge workers frustrated at obsolete and over-engineered search engines. Open source is, of course, popular with those that want to configure their search engine themselves or have a service provider do it and, thus, have a lot of control over how the engine works, as well as the results it delivers. Apache Lucene is also part of many commercial, high-end enterprise search products, including those of IBM.

Midmarket search

Mid-market search is a somewhat vague area, where vendors are succeeding in deals of roughly $75,000-250,000 selling intranet search. This area has thinned out as some vendors have tried to move upmarket into the world of search-based applications, but there are still many vendors making a decent living here. However, SharePoint has had a major effect on this part of the market, and if enterprises already have SharePoint – and Microsoft reckons more than 70% have at least bought a license at some point already – then it can be tough to offer a viable alternative. However, if SharePoint isn’t the main focus, then there is still a decent business to be had offering effective enterprise search, often in specific verticals, albeit without a huge amount of vertical customization.

OEM

The OEM search business has become a lot more interesting recently, in part due to which vendors have left it, leaving space for others. Microsoft’s acquisition of FAST in early 2008 meant one of the two major vendors at the time had essentially left the market entirely, since its focus moved almost entirely to SharePoint, as we recently documented. The other major OEM vendor at the time was Autonomy, and while it would still consider itself to be so, we think much of its OEM business, in fact, comes from document filters, rather than the OEMing of the IDOL search engine. Autonomy would strongly dispute that, but it might be moot soon anyway – it now looks as if it will end up as part of Hewlett-Packard following the announcement of its acquisition at a huge valuation, on August 18.

Those exits have left room for the rise of other vendors in the space. Key markets here include archiving, data-loss prevention and e-discovery. Many tools in these areas have old or quite basic search and text analysis functionality embedded in them, and vendors are looking for more powerful alternatives.

Search-based applications

The high end of the enterprise search market has become, in effect, the market for search-based applications (SBA) – that is, applications that are built on top of a search engine, rather than solely a relational database (although they often work alongside a database). These were touted back in the early 2000s by FAST, but it was too early, and FAST was too complex a set of tools to give the notion widespread acceptance. But in the latter part of the last decade and this one, SBAs have emerged as an answer to the problem of generic intranet search engines getting short shrift from users dissatisfied that the search engines don’t deliver what they want, when they want it.

Until recently, SBAs have mainly been a case of the vendors and their implementation partners building one-off custom applications for customers. But they are now moving to the stage where out-of-the-box user interfaces are being supplied for common tasks. In other words, it’s maturing in a similar way to the application software industry 20 years ago, which was built on top of the explosion in the use of relational databases.

We’ve seen examples in manufacturing, banking and customer service, and one of the key characteristics of SBAs is their ability to combine structured and unstructured data together in a single interface. That was also the goal of earlier efforts to combine search with business-intelligence tools, which often simply took the form of adding a search engine to a BI tool. That was too simplistic, and the idea didn’t really take off, in part because search vendors hadn’t paid enough attention to structure data.

But SBAs, which put much more focus on the indexing process than earlier efforts, appear to be gaining traction. If we were to get to the situation where search indexes are considered a better way of manipulating disparate data types than relational databases, that would be a major shift (see big data). Another key element of successful SBAs is that they don’t look like traditional search engines, with a large amount of white space and a search bar in the middle of the screen. Rather, they make use of facets and other navigation techniques to guide users through information, or often simply to present the relevant information to them.

As I mentioned, there’s more in the full report, including more about specific vendors, total (or big) data and the impact of social media. If you’d like to know more about it, please get in touch with me.