Showing posts with label fast for sharepoint. Show all posts
Showing posts with label fast for sharepoint. Show all posts

Thursday, June 28, 2012

How-to: Get individual items from a delimited field using FS4SP

Thought I’d share a small useful program I often use when indexing data into FAST Search for SharePoint. The issue is that you get data back from some system where the values are concatenated into one field. For example:

user1,user2,user3

But you want to treat each value as a separate one. In FS4SP you have to delimit the values with the unicode character U+2029. The task is simple; replace the exiting delimiter character with this special one.
If you get the data via BCS you might can change the SQL for a view to do this directly(haven’t tested), or you can employ a custom extensibility program. If doing the latter here’s the code:
using System;
using System.Xml;

namespace mAdcOW.MultiValue
{
  class Program
  {
    static int Main(string[] args)
    {
      try
      {
        XmlDocument doc = new XmlDocument();
        doc.Load(args[0]);
        foreach (XmlNode node in doc.SelectNodes("Document/CrawledProperty[@varType='31']"))
        {
          // here you can add replacement for other characters
          // as well besides comma
          node.InnerText = node.InnerText.Replace(',', '\u2029');
        }
        doc.Save(args[1]);

      }
      catch (Exception e)
      {
        Console.WriteLine("Failed: " + e.Message + "/" + e.StackTrace);
        return 1;
      }
      return 0;
    }
  }
}

Thursday, June 21, 2012

Why more people should blog (or My bad day hunting down an error with FS4SP)

Today has been one of those days where you feel you haven’t done anything productive, thinking you have tried all tricks in the book to fix some weird error to no avail.

And just now the solution came to me after spending some time with my family and letting my brain work it’s magic without my focus. And I’m just about 100% sure it will work when I test it tomorrow morning.

The short version

Always use mklink /J when moving a folder from within your FAST installation to another volume (and blog about any error you encounter for others to read).

Wednesday, June 13, 2012

Adding prefix logic to the ribbon search box

A common scenario when upgrading from SharePoint search to FAST Search for SharePoint is that you want to run the new search solution side by side with the old until you are comfortable switching.
image
One solution is having two search centers which targets the old and the new index. In my experience end-users find it cumbersome to navigate to the new search center in order to do testing because they want to use the search box in the ribbon as the entry point to the search center. The chosen solution for this post is to allow users to enter a prefix character in the search box to opt-in to use the new search engine.

Monday, April 16, 2012

When numbers are important to your organization

I was contacted recently by a financial organization which had issues with the sorting of their search results using FAST Search Server 2010 for SharePoint. The issue was that when including year numbers in the search query, they were not given any relevance while ranking the search results.

A typical query could be income statement 2010, but the top results returned documents for 2006.

Clearly the year number 2010 was being excluded from ranking the results, and terms excluded by rank indicate stop word threshold issues.

Thursday, March 29, 2012

The book is out!

If you haven’t already seen my tweets or noticed yourself, “Working with Microsoft FAST Search Server 2010 for SharePoint” is ready for sales in print edition at both Amazon and O’Reilly web sites. The e-book should arrive shortly as well.

It’s been fun and a lot of work creating the book, but together with my co-authors Marcus Johansson and Robert Piddocke we managed to complete it.
I hope and think that everyone should find something interesting and useful in it and I’d love to get your comments and thoughts on it.



I would also like to thank my employer Puzzlepart for supporting my writing and allowing me time during the day for parts of the writing. Very valuable indeed!

Friday, March 9, 2012

How to: Make folder hits appear at the end of your search results

imagePaul Olenick tweeted me a question on how to get folder hits to appear at the end of the search results using FAST Search for SharePoint.

If we are talking about folders from file shares this is not too hard. If we refer to my post about Taking promotions/demotions one step further we can accomplish this by adding negative promotion using XRANK.

Tuesday, December 13, 2011

Returning Best Bets and Visual Best Bets with KeywordQuery and FAST Search for SharePoint

When using the Search Center the Best Bets or Visual Best Bets functionality works like a charm, but I have not been able to get it to work using the KeywordQuery class in SharePoint.

I had done some research previously on the matter after being contacted by Xavier Vanneste who blogged about this (http://blog.xvanneste.com/Lists/Billets/Post.aspx?ID=82), but was not able to figure it out at the time.

I’m currently writing sample code for my upcoming book on FS4SP and I had to figure this out. Time to bring out the tools!

One hour with the Visual Studio Reflector plugin and setting break-points internally in the SSA I finally figured it out. And in a way it was obvious.

Saturday, December 10, 2011

How to prevent an item from being indexed with FAST for SharePoint

Yes, it can be done!

[Update 2012-05-25: There is an even better hack to solve this issue as Kai wrote in a commen below this post. Check out Segei Sergeev answer in this TechNet thread.]

It’s Saturday, my kid has gone to sleep, and I finally have time to tell you guys the good news. Preventing an item from being index, or to paraphrase, to drop a document in the document processing pipeline is indeed possible!

You can already prevent items from being indexed by limiting SharePoint lists and libraries from being crawled with library settings and you can use crawl rules to exclude certain url patterns. But what I am talking about is preventing an item from being indexed based business rules in your organization and looking at the meta data of the item or inside the text of a file.

There are many scenarios for not wanting an item searchable. You might want to prevent indexing items in your organization which contains the super secret codename “wobba”, or items of a certain ContentType. When indexing file shares you don't have much meta data to go on at all for excluding items, so creating your own module with the proper rules might be the only way to go.

Sunday, November 20, 2011

Taking promotions/demotions one step further in FAST Search for SharePoint

promotionsI am not sure how many times I have read the documentation on promotions and demotions in Microsoft documentation, but I can say it is numerous.

And yet yesterday I discovered a valuable piece of nugget which I’m sure I will use a lot in the future.

Friday, October 21, 2011

Error while running the FAST for SharePoint post configuration

During a customer installation the other day my project colleagues encountered a weird issue when configuring up a FAST server. (This was after we got all policy issues resolved due to server hardening – which took some time as well).

When running the configuration wizard they got an unexpected error, and the configuration didn’t finish.
Time to hit the installer log…. The error they got was hidden a bit down the load and was coming from the “ULS Common Core Components” which is installed during configuration.

20.10.2011 14:42:39 Verbose InstallAction - Installing ULSCommonCore services
20.10.2011 14:42:39 Verbose Executing ULSCommonCore installer msiexec /i "C:\FASTSearch\\installer\files\ulscommoncore.msi" /quiet /norestart INSTALLLEVEL=100 - 
20.10.2011 14:42:39 Verbose Utility.Execute - Starting process msiexec with working directory - , write output - False, file - , user - 
20.10.2011 14:42:41 Warning Utility.Execute - Return code for binary msiexec is not 0. This may indicate that binary didn't execute successfully
20.10.2011 14:42:41 Verbose Utility.Execute - Finished executing msiexec
20.10.2011 14:42:41 Error InstallULSCommonCore - An error occurred while executing binary msiexec. Return code is not 0.
20.10.2011 14:42:41 Error Utility.WriteException - Exception -  : Exception - System.Management.Automation.RuntimeException: Error executing ULSCommonCore installer.


Upon my suggestion they kicked off the ULS installer manually and got the following pop-up which gave information for further internet searches.

image

The error is caused by the registry size being too small and has to be increased. Not too obvious from neither the install log nor the pop-up.

Thanks to the below post for giving the resolution on how to increase it :) http://www.geosoft.com/support/knowledge-base/errormessage/Error-An-error-occurred-during-the-installation-of-assembly-Microsoft-VC90-ATL.

Monday, October 17, 2011

A side-by-side approach for upgrading SharePoint search to FAST

search in capeMore and more businesses are upgrading their existing SharePoint 2010 search solutions to FAST Search for SharePoint. Doing so will most likely require scheduled downtime when you make the actual switch.

This can be avoided by cleverly upgrading your existing Search Service Application to a FAST Query SSA.

Want to learn more? Head over to my article at NothingButSharePoint.com for the details - http://pzl.no/qKacHz.

Thursday, August 11, 2011

Why you should use deep refiners with FAST for SharePoint

With FAST for SharePoint you have what is called deep refiners, compared to shallow refiners which the built-in search uses. The difference is that deep refiners guarantee an exact refiner count, and all possible refiner values for your search, while shallow will calculate the counts based on the first 50 hits (default setting which can be adjusted). In addition to deep refiners, FAST for SharePoint also supports shallow refiners.

The first image shows a query using FAST for SharePoint with deep refiners and the second image show the built-in SharePoint Search with shallow refiners.

image

image

With FAST for SharePoint deep refiners are stored in a separate data structure (mostly in memory), optimized for just that; returning the refiners. Think of it as a pre-calculated refiner tree, ready for the picking.

Shallow refiners on the other hand have to be manually constructed by the query server from the results returned. Returning a large result set, looking at the metadata and building the refiners, is both IO and CPU intensive, as it has to retrieve all metadata per item, and then build the refinement result based on that.

Of course, basing the refiners on 50 results is a speedy operation, but at the cost of the accuracy of deep refiners.

So, when you create a refiner with FAST for SharePoint, be sure to tick off the “Deep refiner” checkbox to take advantage of the refiner structure stored within the search index. You will not only get more accurate results, but in most cases they will also return even faster than shallow ones.

PS! There are cases when deep refiners will be slow, particularly if you have very many unique values within a refiner. Then again, if this is the case, you might want to re-think your search solution.

(Reference: http://technet.microsoft.com/en-us/library/gg193929.aspx)

Wednesday, August 10, 2011

Working with Content Collections via the API–Continued (and yet another bug)

Yesterday I wrote about how you could use the admin API to create a new content collection for FAST Search for SharePoint. I explored this further, and discovered a more serious bug. From within a SharePoint context it will never work (see the end of this post for the solution)

Tuesday, August 9, 2011

How-to: Create a content collection in code

(Update: Read http://techmikael.blogspot.com/2011/08/working-with-content-collections-via.html for how to do this in a SharePoint context)


When installing FAST Search for SharePoint it creates a default content collection named “sp” where all content crawled via the FAST Content SSA is stored.

You also have the option to create new collections, and this is typically something you would do for the FAST Specific connectors (FAST Enterprise Web Crawler, FAST Database Connector, FAST Lotus Notes Connector) in order to support management like clearing out all content.

A content collection is merely a logical grouping of content inside of FAST Search for SharePoint, where all items indexed have an additional field named “meta.collection” attached, and not something which affects the physical layout of how FAST stores the search index.

Monday, August 8, 2011

Updated sample deployment files with FAST Search for SharePoint SP1

With service pack 1 (or the June 2011 CU) the sample deployment files included with FAST for SharePoint has been updated. The single server sample is still the same, as there is not much room for changing where you put components, but both the multi-server samples have some changes.
The changes align better with recommend best practice according to the number of nodes used in the samples. Depending on your requirements you can use the provided samples as starting points and modifying them to your needs.

Friday, August 5, 2011

Creating refinement query parameter for FS4SP by code

[Update: 2011-12-09]
Someone from somewhere took the time to use Reflector and pull out the appropriate code from the FS4SP dlls. Download the code below.

When using the Search Center with FAST Search for SharePoint you will see the query parameters being modified when you interact with the page.

The k= parameter holds your search query. This is useful if you create a link somewhere on your site which redirects you to the search page with a pre-set query. An example is creating a link which displays all the documents from a particular employee containing the word contract.

This can be accomplished by the query:


contract author:”Mikael Svenson”

which in the k= parameter looks like:


k=contract%20author%3A%22Mikael%20Svenson%22

And this works just fine. What doesn’t work is if you have specified a synonym for the word contract. When tacking on the author parameter to the query, it will no longer match in the synonym list, as the synonyms match the whole query term, not just a single word.

Sunday, July 24, 2011

Creating a scope to limit by content collection

When working with FAST Search Server for SharePoint and especially if you are using one of the custom FAST connectors you will usually create additional content collections with the command: New-FASTSearchContentCollection
2430680365_410bc05f07_mThis will logically separate content from for example the FAST Enterprise Web Crawler from content indexed from SharePoint into the sp collection (which is the default collection for the FAST Content SSA).

What if you want to show results only from the web crawl? If it had been a content source on the FAST Content SSA you could have created a search scope to limit by content source like this:

New-SPEnterpriseSearchQueryScope -SearchApplication "FAST Query SSA" -Name MyContentScope -Description "Content Source Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter 'contentsource:MyContentSourceName'

Tuesday, July 19, 2011

Released: SharePoint Search Parts

The past couple of weeks I’ve been working evenings and some during the day when my kid is asleep on a FQL (FAST Query Language) enabled Core Results web part. The reason for going fql is not to write it directly into the search box, but in order to provide proper synonym and wildcard support. (Note: fql is only supported with FAST for SharePoint)

imageThe project can be found at http://spsearchparts.codeplex.com/ and is currently in alpha. It might actually be higher quality than what you expect from an alpha version, but I haven’t had time to test it properly.

The biggest undertaking in the project was to write a kql->fql parser. I think the parser works ok at the moment, but there is always room for improvement, and I might have missed something.

The other technical challenge was to enable fql support in the web part. Thankfully, Ivan Neganov has a blog post on how to do this which  I used as foundation for the web part. I did however change the use of direct reflection to use IL.Emit, as it’s more performant and can be cached, but it’s still reflection ;-)

Synonyms are expanded based on single words or phrases. If you add a synonym in SharePoint with two or more words, it will only match in “search phrases” with quotes. I might change this to match multiple terms as well. Synonyms are also lemmatized/stemmed, which they are not in the standard web part.

There is also an option to boost the original query words over the synonyms which could be useful. At least it was fun to implement :-)

Wildcards are supported like they are in fql, where both ? and * can be used, and also anywhere in the words.

When using the web part you can use my query tool at http://fs4splogger.codeplex.com/ to inspect how the kql query is transformed into fql. This will also show synonym expansion and added scopes.

I plan to add more search parts as I have time, and hopefully the project can grow into something really useful.

More information can be found at the project page.

Tuesday, July 12, 2011

Improved relevancy with SP1

This was listed as one of the updated in SP1, and was actually released in the December CU. So what has really changed?

For all the default Metadata Categories in FAST for SharePoint the default behavior is to map auto-discovered crawled properties to the full-text index.

image

This leads to many irrelevant fields being searchable. The update is in fact adding many of these fields to an ignore list, setting the IsMappedToContents property of the crawled property to false.

The full list of the added ignored properties can be found in the function ImproveDefaultSchema in C:\FASTSearch\installer\scripts\include\commontasks.ps1.

The relevancy improvement is to avoid garbage into the index.

Thanks to Microsoft for providing this information.

Saturday, July 2, 2011

SharePoint MVP for 2011

Yesterday was the first day of my July vacation between jobs, and I was spending the evening having a good time with some friends of mine.

I arrived at the restaurant 20 minutes early, ordered a beer and checked my e-mail. And there it was, an e-mail from the Microsoft MVP team reading:

Congratulations 2011 Microsoft MVP

This is my first time being awarded, and I have been awarded in the SharePoint Server category. For those who follow my blog you know that FAST for SharePoint is what I do these days, and it’s a community I very much enjoy being a part of.

Getting an award is something to live up to in my opinion, and I will try my best to continue to do what I love Smile

PS! We all had a blast last night!