Tuesday, February 2, 2016

"I Want to Believe" - Is Microsoft abandoning the enterprise features of enterprise search?

I want to believe that Microsoft has not abandoned the enterprise in enterprise search, and that the good people behind the FAST seearch engine still are allowed to deliver well tested and proven features around the SharePoint search space.

If features are removed, as they were from FAST ESP to FAST Search 2010 from SharePoint, and from FAST Search 2010 for SharePoint to SharePoint 2013, then remove them first after you have tried to advocate the use of them. Just because a feature is not used by the average SharePoint consultant/pro-user, does not mean it’s not valuable. It might be that they just don’t know about it, or don’t know how to put it to good use.

To set the scene for this article, enterprise search is the practice of making content from multiple enterprise-type sources, such as databases and intranets, searchable to a defined audience. And over the years, companies have learned to expect a certain set of features from solutions delivering enterprise search. Wikipedia lists enterprise search having five components:
  • Content awareness
  • Content processing and analytics
  • Indexing
  • Query processing
  • Matching

The present

Before Microsoft dropped FAST ESP and Search Server as separate products they were a player in Gartner’s magic quadrant for Enterprise Search. But as they no longer have stand-alone search offerings, you won’t find them there today. When Microsoft was a part of the quadrant they were in the upper right corner for being both visionary and having the capability to execute. If Microsoft had a separate product in the quadrant today, would they still be in that coveted position?

On the execution part I think yes, as SharePoint 2013 is widely rolled out and SharePoint Online tenants are popping up at a steady pace. Visionary wise, I’m not so sure. With SharePoint 2013 Microsoft gave companies the query rule framework. If you know how to utilize it, the query rule framework is one of the coolest search features out there when developing search-based solutions. When it comes to the indexing and query features coming from FAST ESP, however, we have seen a decline in the feature set. Where FAST ESP gave you full control and FAST Search 2010 for SharePoint locked it down in terms of full customization, SharePoint 2013 has ridden itself of much of the capabilities to tune relevance and control how indexing works. In retrospect, Microsoft seem to have cut some corners on the SharePoint 2013/SPO search features and on the testing of the features that are included.

And don’t be fooled to think Microsoft is not aware of this. There are people out there who care about search functionality, like me, who have in both official and unofficial channels voiced their opinion on the state of Microsoft search within SharePoint.

On-premises features

Don’t get me wrong that it’s all dark ages, because there are some very good features in SharePoint search today that indeed are enterprise grade. I already mentioned the query rule framework, and dictionary extraction and content enrichment via a web service are also prime examples. The enrichment capability makes it a lot easier to do massaging of content to accomplish simple contextual and targeted experiences for the end-user. Could they have been even better? Sure. Those two features are better than FAST Search 2010 for SharePoint offered, but inferior to what FAST ESP offered.

For those not familiar with enriching capabilities, you use them to look at the text and metadata of the content being indexed and then add on even more content to the search index. That could be categorizations or classifications, adding synonyms to disambiguate terms, or applying any business logic you deem necessary.

On the query side, you can still create custom rank profiles used when querying the content to improve the order of how items are returned for users. It’s not for the faint of heart, but once you know how, it’s a very powerful feature – enterprise grade for sure. I would still hold FAST ESP and FAST Search 2010 for SharePoint as winners over the current rank profile offerings, mainly because machine learning plays a part today. Machine learning is hard to comprehend and to tune manually. In theory, machine learning should be superior, if the learning is done correctly.

To sum it up, SharePoint 2013 on-premises and the upcoming SharePoint 2016, which has the exact same feature set as 2013, hits all the five categories for enterprise search mentioned in Wikipedia, but not as well as FAST ESP and FAST Search 2010 for SharePoint did.

Enter hybrid crawl and SharePoint Online

If you decide to take the plunge for the recently hyped hybrid crawl where you index on-premises data into the SharePoint Online search index, or move to SharePoint Online all together, then the story effectively changes. You can no longer do enrichment of the indexed data, and the options for custom rank models are very limited – or useless in my opinion. This might sound harsh, but from a technical perspective, it is what it is.

Effectively point #2 (content processing) and point #5 (matching) of enterprise search have been set back some 10-15 years. Using third-party connector frameworks can mitigate this, but then complexity and cost increase.

Read my blog post Why Hybrid Crawl in SharePoint is a cold hot potato for more thoughts around hybrid crawl.

The future

Where Lucene and Elastic Search-based solutions speak to the developers and allows full flexibility in all parts of the search lifecycle, Microsoft has transformed its FAST search engine from being a flexible kit car to a Prius. I suspect the gasoline engine is still there at its core, but the ease of going electric seems very appealing and friendly to most. Had they gone all-in for a ludicrous mode like a Tesla it would have been a different story.

As of now, if you look to hybrid crawl or SharePoint Online, you do get a cheaper scalable solution, but at the cost of stepping back a decade in functionality. Effectively removing the “enterprise” in enterprise search. Just because you can index a lot of data and have enterprise customers, doesn’t mean it’s enterprise grade search.

Is Office Graph the new messiah?

The Office Graph is definitively a fresh breath into the enterprise search space and the only new search feature launched for SharePoint since SharePoint 2013 entered the market on October 11th, 2012 (and I’m not considering hybrid crawl a search feature per say). To explain in crude terms what the Office Graph does for users, it brings in a new way to rank content based on what you and people like you work with and work on based on recent usage and interaction with content.

Indeed, very useful, although I’m not sure today’s Delve application is the killer user experience to utilize the full potential of the Office Graph. I’ve played with the Office Graph myself, as has others, but I haven’t yet been smart enough to create something useful that I cannot live without on a day-to-day basis.

My Collabograph application, built using the Office Graph, is a fun gizmo playing into the Delve analytics space, but I don’t really need it (and the API has also been discontinued).

That said, Gartner might easily have moved Microsoft up in the visionary quadrant for launching the Office Graph feature, but is it good enough to keep them there in the long run?


Search capabilities in all flavors of SharePoint are still good, and certainly a good contender for enterprise search. Where other solutions lack the wide range of quality, third-party connectors adhering to the source systems strict security control on content access that SharePoint has, SharePoint truly shines. But on the technical side of what you can do with the content during indexing and querying, open source-based solutions and the players in the Gartner magic quadrant looks to have the upper hand today. My wish is that Microsoft should fill in the gaps they forgot back in October of 2012, and also build upon this even further. More capabilities will in general yield more innovation and excellent showcase solutions.

Perhaps the second coming, or comeback if you will, from Microsoft for enterprise grade search is when v2 of the Office Graph programmatic interfaces are made public and we can push all kinds of content and signals into the Office Graph. Or have Microsoft put down their gloves and left the enterprise search arena for good, moving all focus to battle Google and Amazon? That is a large-scale battle with more money to be made for sure.

I’m still a believer in Microsoft as an enterprise search player, and although I have doubts at the current time about the current feature set improving in the near future, I quote Fox Mulder from X-files saying: “I want to believe”.