Monday, May 18, 2015

ALWAYS set the client type on SharePoint search API calls–and be smart about the value

image
At my current customer we recently deployed SharePoint 2013 search, but we don’t seem to get any search suggestions in the search box, thus search-as-you-type in the search box never show anything.

Before diving into the reason behind it all, the solution is to add &clienttype=ContentSearchRegular to all search REST/API calls where you know you will do custom rendering of the result and have no intention to implement the RecordPageClick event when users click on of the result items.

The scenario

When users run search queries and click results, those clicks are recorded back to SharePoint. Once a certain query term have been ran at least six times where an item is clicked, the term is added to the list of query suggestions served by the search box on a search center.

These suggestions help future users to formulate better queries and to get better search results.

The problem

By default a SharePoint tenant will be able to log result clicks for 65.000 items per day. This can be found by the following PowerShell command:
(Get-SPEnterpriseSearchServiceApplication).QueryLogSettings

which will yield the following output:

QLogEnabled              : True
CleanupDays              : 365
LoggingPercent           : 100
TenantDailyLimit         : 65000

Here you see the 65.000 daily limit and the logging percent means it will log all queries coming in. If you set it to 50%, about half of the requests will not be made available for logging.

So what do I mean by made available for logging? When you run a search query a property is returned in the result set named piPageImpression. The value of this property is essential for RecordPageClick to work. If it’s missing no result clicks are logged.

In our SharePoint solution we have in addition to 2013 search also delivered a highly customized search driven intranet. The front page itself have at least five search queries going, and renders the result in a custom UI using JavaScript. There is and was never no intention to feed the clicks of results back from the front page, and it wouldn’t make sense either as the queries are all programmed – not user queries. Being good citizens we have actually added clientype=MyCustomer to all REST calls, but this only allows stats to differentiate the calls, it will not impact using id’s from the piPageImpression daily pool of 65.000 items.

With five calls on the home page alone and 10.000 employees hitting the intranet every time they open their browser, those 65.000 id’s are quickly spent before a single user manage to navigate to the search page and record a proper search click. By the time coffee kicks in for a random employee the FAST search engine is no longer service out piPageImpression id’s and any clicks on results is moot (from a record query suggestion perspective).

The hacky solution

Tell your SP admin to crank up the value of QueryLogSettings to be able to generate more piPageImpression id’s per day.

The proper solution

I first noticed the piPageImpression value was missing for some queries. Next I reflected some SharePoint dll’s and figured piPageImpression was required to record page clicks. So what piece of code generated this value for the search result set? And why did I miss it in the production environment, but was seeing the value in stage, test and development environments? Some more digging led me to the internal Microsoft.Office.Server.Search.Query.QueryProcessor.RecordPageImpression method. A bit down in the function SharePoint checks if client type is 14, 15 or 25, and in those three cases it will NOT generate a piPageImpression id, which is what we want.


If you have used the SharePoint 2013 Query tool you might have noticed a drop down list for Client Type.


image


But how do these names correlate to 14, 15 and 25? Open SQL Management studio, point it to your Search DB and run the following SQL command

select id, name from dbo.MSQLogClientType order by id

and you will get a list of all the names with associated id’s. The description of each one can be found at https://msdn.microsoft.com/en-us/library/ff386740%28v=office.12%29.aspx

  • 14 = ContentSearchHigh
    The search query is sent from a Content Search Web Part where the priority is set to "high".

  • 15 = ContentSearchRegular
    The search query is sent from a Content Search Web Part where the priority is set to "regular".

  • 25 = Monitoring
    This value MUST NOT be used.
Looking at the above we see that the content by search web part is a good boy and uses this already, as you don’t want this web part to be used for generating search suggestions as the web parts themselves have programmatic queries and not user queries.

Since the doc says NOT to use Monitoring, I figured using ContentSearchRegular as your default client type value is the way to go. When you are using the API to search you are in many ways mimicking the CSWP and thus this is the client value to use. If you are creating a custom search result page, then my all means send in something smart like MyCustomer, but then it’s your responsibility to also implement RecordPageClick when a user clicks a result.

Summary

If you are using a lot of search API calls, you total more than 65.000 queries a day, and you are not creating a custom search page which handles RecordPageClick, you should set clienttype=ContentSearchRegular for all your queries!

8 comments:

  1. Good blog post!
    I've developed custom JS that uses the search API to query, and then display search results in a custom HTML (not display templates). So, I'd need to attach the RecordPageClick event. I cannot find any good documentation, do you know how? Could only find: https://msdn.microsoft.com/en-us/library/hh630829(v=office.12).aspx

    ReplyDelete
    Replies
    1. There aren't any :) Use fiddler or similar and copy the same logic. The calls are not that hard to decipher....and haven't implemented it myself yet.

      Delete
    2. Hello Mikael,

      Very good blog post, thanks!

      I'm designing Custom search box with results page using REST api and wondering have you managed to implement RecordPageClick using JS.

      Roland - Have you had any luck implement RecordPageClick on Custom results page.

      Regards

      Delete
    3. I have done it with CSOM, so should be analog using JSOM. Probably not possible using REST.

      Delete
    4. Hi Ronald,

      Have you implemented this, can you please share your findings?

      Thanks

      Delete
  2. Thanks Mikael, can you post reference to JSOM (RecordPageClick) implementation if possible.

    ReplyDelete
    Replies
    1. I don't have the time, but should be analog to the classes in CSOM. The function is defined in sp.search.js in Microsoft.SharePoint.Client.Search.Query.SearchExecutor.recordPageClick. Taking the same 9 parameters as defined for https://msdn.microsoft.com/en-us/library/microsoft.sharepoint.client.search.query.searchexecutor.recordpageclick.aspx

      Delete