Thursday, August 11, 2011

Why you should use deep refiners with FAST for SharePoint

With FAST for SharePoint you have what is called deep refiners, compared to shallow refiners which the built-in search uses. The difference is that deep refiners guarantee an exact refiner count, and all possible refiner values for your search, while shallow will calculate the counts based on the first 50 hits (default setting which can be adjusted). In addition to deep refiners, FAST for SharePoint also supports shallow refiners.

The first image shows a query using FAST for SharePoint with deep refiners and the second image show the built-in SharePoint Search with shallow refiners.

image

image

With FAST for SharePoint deep refiners are stored in a separate data structure (mostly in memory), optimized for just that; returning the refiners. Think of it as a pre-calculated refiner tree, ready for the picking.

Shallow refiners on the other hand have to be manually constructed by the query server from the results returned. Returning a large result set, looking at the metadata and building the refiners, is both IO and CPU intensive, as it has to retrieve all metadata per item, and then build the refinement result based on that.

Of course, basing the refiners on 50 results is a speedy operation, but at the cost of the accuracy of deep refiners.

So, when you create a refiner with FAST for SharePoint, be sure to tick off the “Deep refiner” checkbox to take advantage of the refiner structure stored within the search index. You will not only get more accurate results, but in most cases they will also return even faster than shallow ones.

PS! There are cases when deep refiners will be slow, particularly if you have very many unique values within a refiner. Then again, if this is the case, you might want to re-think your search solution.

(Reference: http://technet.microsoft.com/en-us/library/gg193929.aspx)

14 comments:

  1. Yes, you are right .
    I have created managed property without check the (deep refiner) and check the result and it was not like total number and then checked with (deep refiner) and it was same as total number ^-^.

    ReplyDelete
  2. After enabling deep refinement you have to reindex your content.

    ReplyDelete
  3. Mikael, I'm trying to create a custom deep refiner on the ows_contenttype crawled property by creating my own managed property mapped to that. I'm seeing the data come through after a full crawl, but when I try to create a custom filter for the filter category, it won't show my custom categories.

    I saw this link on MSDN where someone commented at the end that String types and the ValueMapping MappingType cannot be used in custom properties for FAST.
    http://msdn.microsoft.com/en-us/library/ff625185

    Any idea if that's what could be causing the problem?

    ReplyDelete
    Replies
    1. Hi,
      As long as you are registering your refiner without custom valuemapping in the refiner config it should show. Be sure to set the refinement webpart to not use the "location config" and also set the number of refiners to display to be high enough number to include your custom one if you added it to the bottom.

      A good way to check if the refiner is working is using my logger tool fs4splogger.codeplex.com and monitor what refinement xml is sent out from FAST. If the values are there, then it's a config error on your page. If the data is not there, then the error is in the managed property settings or mapping.

      And make sure your mp has both Queryable and RefinementEnabled set to true.

      As for valuemapping support in the refinement config, I will add a webpart to spsearchparts.codeplex.com when I have time which supports this for FAST.

      Delete
  4. Hi Mikael,

    Thank you for the wonderful information on Deep Refiners.
    All our manged properties used in the refinement panel are deep refiners, we ran a full crawl, but still the refiner count is not properly working. Can you please help us to get out of this problem?

    Thanks,
    Srimanth

    ReplyDelete
    Replies
    1. Hi,
      If you have a lot of content and run queries which hit a lot of them, then the number might be incorrect as it will cut scanning items after some time as well if I remember correctly to make sure performance is not impacted too much.

      Also make sure to turn off duplicate trimming to get more accurate results.

      So, what errors are you seeing in the refiner counts?

      Delete
    2. Thank you for your response.

      Actually, we are not seeing any errors, but the refiner count is incorrectly displayed.

      We tried to uncheck the "Remove Duplicate Results" option from the core results web part, but still the refiner count is not correctly shown.

      Even if the total result count is less than 50 also, the refiner count is shown incorrect.

      Please let us know the root cause of the issue.

      Thanks,
      Srimanth

      Delete
    3. Hi,
      It would be helpful if you could give some information on how you check if the count is accurate or not.

      thanks,
      Mikael

      Delete
    4. let's say if count 3 is shown to a refiner, clicking on that refiner is generating 5 items in the search results instead of 3.

      Delete
    5. Hi,
      with only 3 it seems weird. I would create a new managed property, mark it as refinable(deep), map the cp for it, do a full crawl and check the value. Also, if you have duplicate trimming turned on, are results 3 or 5 in the result list?

      Delete
    6. All our managed properties are deep refiners and also we have done full crawl. With and without duplicate trimming turned on, the results are still misleading i.e., the count is shown wrong.

      Thanks,
      Srimanth

      Delete
    7. How many items all together? And I would still create a new mp just to test this with a different mp.

      Delete
    8. Our refinement panel has 13 refiners i.e., managed properties. I even tested with one refiner, still it is not showing correct refiner count.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete