Tuesday, July 19, 2011

Released: SharePoint Search Parts

The past couple of weeks I’ve been working evenings and some during the day when my kid is asleep on a FQL (FAST Query Language) enabled Core Results web part. The reason for going fql is not to write it directly into the search box, but in order to provide proper synonym and wildcard support. (Note: fql is only supported with FAST for SharePoint)

imageThe project can be found at http://spsearchparts.codeplex.com/ and is currently in alpha. It might actually be higher quality than what you expect from an alpha version, but I haven’t had time to test it properly.

The biggest undertaking in the project was to write a kql->fql parser. I think the parser works ok at the moment, but there is always room for improvement, and I might have missed something.

The other technical challenge was to enable fql support in the web part. Thankfully, Ivan Neganov has a blog post on how to do this which  I used as foundation for the web part. I did however change the use of direct reflection to use IL.Emit, as it’s more performant and can be cached, but it’s still reflection ;-)

Synonyms are expanded based on single words or phrases. If you add a synonym in SharePoint with two or more words, it will only match in “search phrases” with quotes. I might change this to match multiple terms as well. Synonyms are also lemmatized/stemmed, which they are not in the standard web part.

There is also an option to boost the original query words over the synonyms which could be useful. At least it was fun to implement :-)

Wildcards are supported like they are in fql, where both ? and * can be used, and also anywhere in the words.

When using the web part you can use my query tool at http://fs4splogger.codeplex.com/ to inspect how the kql query is transformed into fql. This will also show synonym expansion and added scopes.

I plan to add more search parts as I have time, and hopefully the project can grow into something really useful.

More information can be found at the project page.

21 comments:

  1. Hi Mike,

    We are running into an issue where we have extended the OOB coresearchresults webpart to use FQL & now our "did you mean" are screwed up..
    The did you means come back as "Did you mean? string("abc",mode="simpleall",annotation_class="user")" with the FQL part as hyper link.

    Have you happen to see this kind of behaviour & do you have any suggestions to counter this ?

    Any help would be greatly appreciated!

    Thanks,
    Ashwani

    ReplyDelete
    Replies
    1. Haven't seen it, but I'm not surprised. You can create a scheduled job to run over all suggestions, and remove the ones which are FQL.

      Should be quite easy with PowerShell.

      Delete
    2. I am not sure if I follow this...
      Are you suggesting that spell check suggestions are stored somewhere & can be cleaned up ?
      May be I am missing something here, just to be clear the issue is not with Query Suggestion(autocomplete). Its with spell check i.e. Did you mean?
      However, even if I am able to do that, this would not solve the problem.
      The Did you Mean webpart on the SP search results page would continue to show did you mean with FQL on it.
      Is that expected or that can be changed without extending the did you mean webpart ?

      Delete
    3. Hi, I mis-read it, but there are options to be made.
      Check out http://technet.microsoft.com/en-us/library/ff393791.aspx for details on the spelltuning.

      That said, go to the Query SSA -> FAST Admin -> Spell checking management.

      There you can add words which should be ignored for spell-checking. That might help you.

      And the reason it returns the full fql, is that the fql is being interpreted as the user input. It might actually be easier to filter this in the UI, excluding "did you mean" which are fql.

      So using FQL like this certainly has some negative sides to it it seems :(

      Delete
    4. Thanks Mike..
      We do not really want to ignore any spell-check terms.
      Its about displaying them in a way that looks clean.
      We can filter the FQL in the UI, but that would be a bit too much coz we will still have to ensure that the next time the user clicks on the suggestion it goes as n FQL.
      We are using FQL coz we want to do some boosting based on managed properties.

      I do understand the fact that FQL is interpreted as user input. I guess we tried to do a quick fix by just overriding the CoreResults web & enabling FQL on it.
      Seems like this may not be as simple as it appears to be.

      Delete
    5. Hi,
      If your FQL statements are static, meaning you have a list of statements added based on rules, you can use scopes instead. Create all your scopes with the FQL, then add scopes to the query using KQL instead based on your rules.

      Delete
    6. Hi, the FQL values are not static. Those are based on the logged in users language & country.

      Delete
    7. That can be made static.

      if( user.language = "no" ) AddScope("Norwegian")
      if( user.country = "nl" ) AddScope("Netherlands")

      This works as long as whatever statement you want to apply on the query doesn't change, so that it can be entered into a scope.

      Delete
    8. Sure, we can do that.
      But that would make around 70 scopes for me as i have 7 content sources & 10 languages/countries. It would be a bit too much to maintain..

      And then my query is a not totally static to put in a scope.
      Here's my query:
      string("{0}",mode="simpleall",annotation_class="user") and xrank(string("{0}"),country:string("{1}"),boost=10000) and filter(sourcename:string("XYZ"))

      Basically, we have to do boosting & ensure that we get all the keyword mgmt features available on the search center. Which otherwise go off if we use just the XRANK query.

      Thanks,
      Ashwani

      Delete
    9. I've done a scope solution with around 30 scopes before which was autogenerated with a job. So management is not that hard. And adding the scope programatically is pretty much equal to hand crafting fql.

      Also your query seems "static" enough from what you write. Using scopes would also make keywords/bestbet/synonyms work out of the box.

      But, there are always limitations and we have to make do with the tools at hand. Good luck with your solution :D

      Delete
    10. Sure Mike.
      I don't think we want to go the route of maintaining 70 scopes for this project. Our env is shared across all the search implementations in the organization & there are quite a few of them there.

      For now, we have added another CoreResults webpart on the page in hidden mode & set the cross webpart QueryID for the search summary webpart & the hidden one as same.
      So basically we are serving did you mean with a separate query without FQL.
      Not the best of the solutions, but we can live with the additional query load this would generate.

      Thanks,
      Ashwani

      Delete
    11. Hi,
      I think that's a smart and creative solution :)

      Delete
  2. Hi Mike,
    I’m using search.asmx to fetch results from FAST backend using WCF application which is deployed in a sharepoint (2010) site collection. In the queryXml I am not passing any synonyms but at the backend, query synonyms are appending by default. I don’t want synonyms to be appended in my back end query. I had tried the following methods but no luck…
    • Passing “qtf_querysynonyms=false” as a parameter in the FQL query.
    String(“hand”) AND qtf_querysynonyms=false

    • Passing “qtf_querysynonyms=false” as a string operator in the FQL query.
    String(“hand”, qtf_querysynonyms=false)
    Moreover I am unable to see “qtf_querysynonyms” parameter at back end query log and also “Query Transform” element doesn’t have “synonyms”.
    Can you please let me know how to stop default synonym expansion and how to enable “synonyms” in “Query Transform” section?
    Thanks in advance……

    ReplyDelete
    Replies
    1. Hi,
      You cannot stop synonym expansion with fql parameters.

      Synonyms are local to the site collection, so the easiest would be to execute your searches against a site collection where you don't have any keywords added.

      Synonyms are added at http://contoso.com/_layouts/contextualkeywordmanagement.aspx

      Delete
  3. Hi Mike,
    Thanks for your reply.
    I have few more questions, Can you please clarify those questions as well….
    • Many blogs says that synonym expansion can be turned off by using "qtf_querysynonyms=false", if it is true can you tell us how to execute this.
    • QueryEx query elements has "EnableStemming" element for enabling lemmatization. Does it have any element for disabling synonym expansion as well?

    If above two ways don’t work, can you please tell me, is there any other way to stop the query side synonym expansion.
    Thanks in advance.........

    ReplyDelete
    Replies
    1. Hi,
      Afaik that parameter is only valid for FAST ESP. Like I said, the easiest is to query against another site collection or FASTSearchSearchSettingGroup.

      From page 222 in my book:
      ---
      More Info The search settings group name is in the form of a GUID and is a reference
      to the site collection for which your keywords should be added. From a SharePoint
      Management Shell, execute the following command to get the correct GUID.
      (Get-SPSite -Identity "http://test").ID
      Then in an FS4SP shell, create the new search settings group by using the following command.
      Use the GUID from the previous command as the name parameter if it is not already
      created for you.
      New-FASTSearchSearchSettingGroup –Name ""
      ---

      Basically, create a new group for a new site collection or an existing one which does not have keywords defined. Then query against this group on your queries by specifying the search group UserContextGroupID. Setting this property is however not available via the web service afaik (http://msdn.microsoft.com/en-us/library/ms563775(v=office.14).aspx). But it might just work to query on that new site collection which has a new search group with the same guid.

      Delete
  4. Hi Mikael,

    We have a requirement to enable synonym as well as use "a" parameter in the URL. I used the FQLCoreResult webpart that you have shared but it does not support "a" parameter in the url. Can you please explain how we can use "a" parameter.

    Thanks

    ReplyDelete
    Replies
    1. Hi,
      Short answer, modify the code :) I logged it as an issue a long time ago, but haven't had time or need to work on the code.

      http://spsearchparts.codeplex.com/workitem/21229

      Thanks,
      Mikael Svenson

      Delete
  5. Hi Mike,

    How does the marriage of FQL & Synonyms work in this case on the searchparts ?
    If the site collection has synonyms configured, these search parts use those synonyms to add to the FQL query... correct ?
    If yes, how does it disable the addition of the synonyms on the overall query, i thought the configured synonyms would be pulled up & sent to FS4SP ?
    or may be i am not understanding the way synonyms would work with a webpart with enableFQL=true ?

    so for an example (red car being the synonym) where i expanded the synonym in the FQL like and (or("car", "red car"), filter(site:http://xyz)) the final FQL that gets executed might be or((or("car", "red car"), filter(site:http://xyz)), "red car").. no ?

    can you please help clarify ?

    thanks,
    Ashwani

    ReplyDelete
    Replies
    1. Hi,
      In order for a synonym to match the query part will have to match that of the synonym. If you add a synonym with the matching part:

      or("car, "red car")

      then it will match :) So it matches the text coming in, regardless of the query mode. When you write FQL yourself it's almost impossible to match a keyword entry :)

      Delete
    2. Sure... I got it..
      My confusion is that you are reading the Synonyms from the searchSettingsGroup of the site where the webparts are hosted...
      This site has Synonyms configured... will the system not add the synonym to the query that you are making up by default at run time in addition to the FQL query you are creating by adding the synonym in the FQL in the code ?
      Not able to understand how that is being skipped, if at all ?

      Delete