My musings on Office 365, search, programming and technology.
(June 1st, 2019, I started as a full time employee of Microsoft, thus any post before that date are solely written on my own behalf)
Tuesday, December 13, 2011
Returning Best Bets and Visual Best Bets with KeywordQuery and FAST Search for SharePoint
I had done some research previously on the matter after being contacted by Xavier Vanneste who blogged about this (http://blog.xvanneste.com/Lists/Billets/Post.aspx?ID=82), but was not able to figure it out at the time.
I’m currently writing sample code for my upcoming book on FS4SP and I had to figure this out. Time to bring out the tools!
One hour with the Visual Studio Reflector plugin and setting break-points internally in the SSA I finally figured it out. And in a way it was obvious.
Saturday, December 10, 2011
How to prevent an item from being indexed with FAST for SharePoint
Yes, it can be done!
[Update 2012-05-25: There is an even better hack to solve this issue as Kai wrote in a commen below this post. Check out Segei Sergeev answer in this TechNet thread.]It’s Saturday, my kid has gone to sleep, and I finally have time to tell you guys the good news. Preventing an item from being index, or to paraphrase, to drop a document in the document processing pipeline is indeed possible!
You can already prevent items from being indexed by limiting SharePoint lists and libraries from being crawled with library settings and you can use crawl rules to exclude certain url patterns. But what I am talking about is preventing an item from being indexed based business rules in your organization and looking at the meta data of the item or inside the text of a file.
There are many scenarios for not wanting an item searchable. You might want to prevent indexing items in your organization which contains the super secret codename “wobba”, or items of a certain ContentType. When indexing file shares you don't have much meta data to go on at all for excluding items, so creating your own module with the proper rules might be the only way to go.
Monday, November 28, 2011
Creating a Secure Store Target application with access for all users
Instead of storing the logon information of the service account in a list or web.config we want to store this in Secure Store which is ideal for storing credentials (big surprise there). We also want to create the Secure Store target application in code with an installer and not using Central Admin.
Sunday, November 20, 2011
Taking promotions/demotions one step further in FAST Search for SharePoint
Friday, October 21, 2011
Error while running the FAST for SharePoint post configuration
When running the configuration wizard they got an unexpected error, and the configuration didn’t finish.
Time to hit the installer log…. The error they got was hidden a bit down the load and was coming from the “ULS Common Core Components” which is installed during configuration.
20.10.2011 14:42:39 Verbose InstallAction - Installing ULSCommonCore services 20.10.2011 14:42:39 Verbose Executing ULSCommonCore installer msiexec /i "C:\FASTSearch\\installer\files\ulscommoncore.msi" /quiet /norestart INSTALLLEVEL=100 - 20.10.2011 14:42:39 Verbose Utility.Execute - Starting process msiexec with working directory - , write output - False, file - , user - 20.10.2011 14:42:41 Warning Utility.Execute - Return code for binary msiexec is not 0. This may indicate that binary didn't execute successfully 20.10.2011 14:42:41 Verbose Utility.Execute - Finished executing msiexec 20.10.2011 14:42:41 Error InstallULSCommonCore - An error occurred while executing binary msiexec. Return code is not 0. 20.10.2011 14:42:41 Error Utility.WriteException - Exception - : Exception - System.Management.Automation.RuntimeException: Error executing ULSCommonCore installer.
Upon my suggestion they kicked off the ULS installer manually and got the following pop-up which gave information for further internet searches.
The error is caused by the registry size being too small and has to be increased. Not too obvious from neither the install log nor the pop-up.
Thanks to the below post for giving the resolution on how to increase it :) http://www.geosoft.com/support/knowledge-base/errormessage/Error-An-error-occurred-during-the-installation-of-assembly-Microsoft-VC90-ATL.
Monday, October 17, 2011
A side-by-side approach for upgrading SharePoint search to FAST
More and more businesses are upgrading their existing SharePoint 2010 search solutions to FAST Search for SharePoint. Doing so will most likely require scheduled downtime when you make the actual switch.
This can be avoided by cleverly upgrading your existing Search Service Application to a FAST Query SSA.
Want to learn more? Head over to my article at NothingButSharePoint.com for the details - http://pzl.no/qKacHz.
Wednesday, September 21, 2011
NSC Talk–SharePoint search planning 101
The first talk was by Henning Strand from Mills. Henning talked about their process portal to support product development, a portal made with very little code. The second talk was by my colleague Mads Nissen, outlining how we at Puzzlepart do apps development for SharePoint without using SharePoint – only ASP.Net and Excel. Both talks were very interesting and targeted both end-users and devs.
Last one out was myself doing a slide only talk on SharePoint search planning, where I tried to level the crowd from level 100 to level 400 on the SharePoint maturity model regarding search, in just 25 minutes.
Henning and Mads both went over their allotted time so I did my presentation in a lightning fast tempo. My Cliffs Notes reference went above everyone's heads as they had never heard about it, but I hope the crowd got the message in spite of me talking extremely fast. I talk quite fast when getting excited, and trying to catch up the lost time didn’t help :)
Also thanks to Michal Pisarek for letting me "steal" from “Why Your SharePoint Search Sucks”, a slide deck worth going over. Mine was sort of the Cliffs Notes of his talk. Both talks about the same topic but with different approaches.
Tuesday, September 6, 2011
How would you optimize this piece of C# code?
if (Math.Max(thisString.Length, otherString.Length) > Math.Pow(2, 31)) throw new ArgumentException("String too long");
Looking at the code I see right away that this is ported from some other language over to C#. It’s quite easy to optimize it…. how would you do it?
Tuesday, August 30, 2011
Adding the left navigation menu to web part pages in SharePoint 2010
This is nothing new and it has been blogged about since 2007. But on my 15 minute Google spree this morning I could not find a recipe which worked 100% with SharePoint 2010, so I will give it a try myself, and hopefully this can help some others in the future. I’m sure someone else has a complete post on this which I missed, but at least I have leveled up myself in the use of SharePoint Designer and modifying existing layouts instead of creating custom ones :)
Thursday, August 11, 2011
Why you should use deep refiners with FAST for SharePoint
With FAST for SharePoint you have what is called deep refiners, compared to shallow refiners which the built-in search uses. The difference is that deep refiners guarantee an exact refiner count, and all possible refiner values for your search, while shallow will calculate the counts based on the first 50 hits (default setting which can be adjusted). In addition to deep refiners, FAST for SharePoint also supports shallow refiners.
The first image shows a query using FAST for SharePoint with deep refiners and the second image show the built-in SharePoint Search with shallow refiners.
With FAST for SharePoint deep refiners are stored in a separate data structure (mostly in memory), optimized for just that; returning the refiners. Think of it as a pre-calculated refiner tree, ready for the picking.
Shallow refiners on the other hand have to be manually constructed by the query server from the results returned. Returning a large result set, looking at the metadata and building the refiners, is both IO and CPU intensive, as it has to retrieve all metadata per item, and then build the refinement result based on that.
Of course, basing the refiners on 50 results is a speedy operation, but at the cost of the accuracy of deep refiners.
So, when you create a refiner with FAST for SharePoint, be sure to tick off the “Deep refiner” checkbox to take advantage of the refiner structure stored within the search index. You will not only get more accurate results, but in most cases they will also return even faster than shallow ones.
PS! There are cases when deep refiners will be slow, particularly if you have very many unique values within a refiner. Then again, if this is the case, you might want to re-think your search solution.
(Reference: http://technet.microsoft.com/en-us/library/gg193929.aspx)
Wednesday, August 10, 2011
Working with Content Collections via the API–Continued (and yet another bug)
Tuesday, August 9, 2011
How-to: Create a content collection in code
When installing FAST Search for SharePoint it creates a default content collection named “sp” where all content crawled via the FAST Content SSA is stored.
You also have the option to create new collections, and this is typically something you would do for the FAST Specific connectors (FAST Enterprise Web Crawler, FAST Database Connector, FAST Lotus Notes Connector) in order to support management like clearing out all content.
A content collection is merely a logical grouping of content inside of FAST Search for SharePoint, where all items indexed have an additional field named “meta.collection” attached, and not something which affects the physical layout of how FAST stores the search index.
Monday, August 8, 2011
Updated sample deployment files with FAST Search for SharePoint SP1
The changes align better with recommend best practice according to the number of nodes used in the samples. Depending on your requirements you can use the provided samples as starting points and modifying them to your needs.
Friday, August 5, 2011
Creating refinement query parameter for FS4SP by code
Someone from somewhere took the time to use Reflector and pull out the appropriate code from the FS4SP dlls. Download the code below.
When using the Search Center with FAST Search for SharePoint you will see the query parameters being modified when you interact with the page.
The k= parameter holds your search query. This is useful if you create a link somewhere on your site which redirects you to the search page with a pre-set query. An example is creating a link which displays all the documents from a particular employee containing the word contract.
This can be accomplished by the query:
contract author:”Mikael Svenson”
which in the k= parameter looks like:
k=contract%20author%3A%22Mikael%20Svenson%22
And this works just fine. What doesn’t work is if you have specified a synonym for the word contract. When tacking on the author parameter to the query, it will no longer match in the synonym list, as the synonyms match the whole query term, not just a single word.
Sunday, July 24, 2011
Creating a scope to limit by content collection
What if you want to show results only from the web crawl? If it had been a content source on the FAST Content SSA you could have created a search scope to limit by content source like this:
New-SPEnterpriseSearchQueryScope -SearchApplication "FAST Query SSA" -Name MyContentScope -Description "Content Source Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter 'contentsource:MyContentSourceName'
Tuesday, July 19, 2011
Released: SharePoint Search Parts
The past couple of weeks I’ve been working evenings and some during the day when my kid is asleep on a FQL (FAST Query Language) enabled Core Results web part. The reason for going fql is not to write it directly into the search box, but in order to provide proper synonym and wildcard support. (Note: fql is only supported with FAST for SharePoint)
The project can be found at http://spsearchparts.codeplex.com/ and is currently in alpha. It might actually be higher quality than what you expect from an alpha version, but I haven’t had time to test it properly.
The biggest undertaking in the project was to write a kql->fql parser. I think the parser works ok at the moment, but there is always room for improvement, and I might have missed something.
The other technical challenge was to enable fql support in the web part. Thankfully, Ivan Neganov has a blog post on how to do this which I used as foundation for the web part. I did however change the use of direct reflection to use IL.Emit, as it’s more performant and can be cached, but it’s still reflection ;-)
Synonyms are expanded based on single words or phrases. If you add a synonym in SharePoint with two or more words, it will only match in “search phrases” with quotes. I might change this to match multiple terms as well. Synonyms are also lemmatized/stemmed, which they are not in the standard web part.
There is also an option to boost the original query words over the synonyms which could be useful. At least it was fun to implement :-)
Wildcards are supported like they are in fql, where both ? and * can be used, and also anywhere in the words.
When using the web part you can use my query tool at http://fs4splogger.codeplex.com/ to inspect how the kql query is transformed into fql. This will also show synonym expansion and added scopes.
I plan to add more search parts as I have time, and hopefully the project can grow into something really useful.
More information can be found at the project page.
Tuesday, July 12, 2011
Improved relevancy with SP1
This was listed as one of the updated in SP1, and was actually released in the December CU. So what has really changed?
For all the default Metadata Categories in FAST for SharePoint the default behavior is to map auto-discovered crawled properties to the full-text index.
This leads to many irrelevant fields being searchable. The update is in fact adding many of these fields to an ignore list, setting the IsMappedToContents property of the crawled property to false.
The full list of the added ignored properties can be found in the function ImproveDefaultSchema in C:\FASTSearch\installer\scripts\include\commontasks.ps1.
The relevancy improvement is to avoid garbage into the index.
Thanks to Microsoft for providing this information.
Saturday, July 2, 2011
SharePoint MVP for 2011
Yesterday was the first day of my July vacation between jobs, and I was spending the evening having a good time with some friends of mine.
I arrived at the restaurant 20 minutes early, ordered a beer and checked my e-mail. And there it was, an e-mail from the Microsoft MVP team reading:
Congratulations 2011 Microsoft MVP
This is my first time being awarded, and I have been awarded in the SharePoint Server category. For those who follow my blog you know that FAST for SharePoint is what I do these days, and it’s a community I very much enjoy being a part of.
Getting an award is something to live up to in my opinion, and I will try my best to continue to do what I love
PS! We all had a blast last night!
Wednesday, June 29, 2011
FAST for SharePoint Service Pack 1
The most notable fixes for me are:
- Improves default schema that improves relevancy
- Better title and date extraction for Microsoft Office documents
- Adds more flexible custom property extractors
- Updated sample deployment templates <- Note: My example deployment templates were messed up during the install of SP1
- Managed properties does no longer have to be in lowercase in the refinement panel configuration, in the query API’s or in FQL scope filters
- Improves index backup and index restore
- Improved indexing speed for several scenarios
- Fixed index corruption when the indexer crashes
Functionality to add search columns to a live system[Edit: Was not included afterall, but will in a later CU]- Infotool.exe will pick up more log files and configuration files
In this new file you can add as many extractor dictionaries as you want, not just three whole-word and two word-part dictionaries as was the limit before. And you can also have both case and case-insensitive matching.
There is a migration guide at: Migrate custom extractor to sp1
Friday, June 24, 2011
Error in default important level weights for the Full-text index mappings
First of all, the relevance score for a search hit in FAST for SharePoint is build up of many different values, where one part of the score is how important is the field which contains data where your query matched. As an example if your query matches words in the title it will get rated higher compared to if it matched in the body text of the document.
Wednesday, June 22, 2011
Deleting a Federated Location via the API
To accomplish this I use the Microsoft.Office.Server.Search.Administration.SearchServiceApplication class, and more specifically the AddNewLocationConfiguration and DeleteLocationConfiguration methods. I tried to use the same methods of the SearchServiceApplicationProxy first, but there seems to be an error using the AddNewLocationConfiguration method on proxy object.
AddNewLocationConfiguration takes a LocationConfiguration object as parameter, which can be created by loading in the xml from an OSDX file like this:
var searchLocation = new LocationConfiguration();
var stream = …stream to xml file…; searchLocation.Import(stream);
As you see, adding is not too hard, but deleting is the problem. The DeleteLocationConfiguration method takes the id of a LocationConfiguration object as the parameter. But lo and behold, the Id property is defined as internal.
Friday, June 17, 2011
Making Best Bets available right away
This is easily fixed by running the “Prepare query suggestions” timer job. Seems this jobs handles anything to do with synonyms, nicknames and also keywords.
In a SharePoint Management PowerShell window you can start the timer job by issuing
Start-SPTimerJob -Identity "Prepare query suggestions"
Wednesday, June 15, 2011
Clearing items from a specific content source
For FAST Search for SharePoint you would have to call Clear-FASTSearchContentCollection on the FS4SP farm as well via PowerShell.
This is all fine if you want to remove everything, but sometimes you may want to only remove parts of the index, say only items from a particular content source like SharePoint or your file server.
Thursday, June 2, 2011
DateTime resolution on your SharePoint Search Center hits
Saturday, May 28, 2011
Where did those search results go?
Index cleanup
The index cleanup stage occurs when you delete a content source or start address, or both, from a search service application. It can also occur when the content indexing connector cannot find a host supplying content: The indexing connector will look for the host during three consecutive crawls, but if the host is not found it will delete the content source and cause the index to enter the cleanup stage.
Saturday, May 14, 2011
Document Thumbnails and PowerPoint Preview for your search results without installing FAST for SharePoint
Wouldn’t it be nice to have this feature on all versions of SharePoint? I will tell you how, as I demonstrated in a proof of concept talk at Arctic SharePoint Challenge 2011.
[Update June 7th 2011 - Check out Thumbnail Extender for SharePoint on CodePlex for a thumbnail webpart using OWA]
Thursday, May 5, 2011
FAST for SharePoint Query Logger v2
I just finished off v2 of the FAST for SharePoint Query Logger and the download is available from http://fs4splogger.codeplex.com/.
New functionality added is:
- Pretty print of the executed FQL
- List any refiners used on the query
- Display the rank calculation per entry in the result
Thanks to FAST for providing a script to show the calculations, which I ported and modified for this program.
Wednesday, May 4, 2011
Working with crawled and managed properties via code
Instead of having a deployment script creating your properties, you could have an event receiver which creates your mappings once a site or list is provisioned. Or when some other action is going on.
So how do you go about doing this?
Monday, May 2, 2011
Three Main Reasons Why You Should Upgrade to FAST for SharePoint
You can read the article over at SharePoint Magazine, and if you like it feel free to click the Facebook Like button at the end.
The other two finalists are:
“Best Practices for SharePoint Groups” by Josh McCarty and “A Guide to Leaving Lotus Notes and Moving to Microsoft SharePoint” by Andrew Vevers.
And if you are serious about search within SharePoint, go FAST!
Thursday, April 28, 2011
Adding new nicknames to SharePoint People Search
But what if you are located in a different part of the world, speaking some obscure language like Norwegian, which is my native tongue, and where names are sounding nothing like English ones? What then?
Thursday, April 14, 2011
DateTime resolution for querying FAST for SharePoint
Tuesday, April 5, 2011
Remember to give your SharePoint 2010 Information Worker Virtual Machine a fresh certificate
Some days ago this certificate expired within the Microsoft Information Worker VM. The reason I noticed is because I’m currently conduction a FAST for SharePoint workshop in Brussels, and this very morning all the attendees got the same error
Failed to connect to demo2010a.contoso.com:13391 Failed to initialize session with document engine: Unable to resolve Contentdistributor
Tuesday, March 29, 2011
Why you still should prefer PowerShell over UI to create Search Scopes in FAST for SharePoint
Tuesday, March 15, 2011
Using FFDDumper to log items in a custom pipeline with FS4SP
There is also a documented way to log all content called FFDDumper which executes as the first step in the processing pipeline.
You enable this stage by editing C:\FASTSearch\etc\config_data\DocumentProcessor\optionalprocessing.xml.
Thursday, March 3, 2011
Microsoft Virtual Labs for Search
Interested in learning more about how to develop great search solutions and what SharePoint 2010 can offer in terms of search capabilities?
Head over to the SharePoint Server Virtual Labs page at MSDN and check out the newly released labs on search.
The labs are virtual machines accessed via your browser and are accompanied by a script to walk you thru the exercises.
Note that some of the labs seem to be missing indexed content, so you might need to kick off a crawl before starting. The machines are also quite slow, and you might have better luck running the “2010 Information Worker Demonstration and Evaluation Virtual Machine” locally instead. It’s more or less the same machine as provided in the labs.
Wednesday, March 2, 2011
Prototyping pipeline stages in PowerShell
To the rescue comes PowerShell. Since all FAST servers have PowerShell installed you can create a PowerShell script and use this. All you need is notepad
Tuesday, March 1, 2011
How not to name a configuration setting
Recently while looking over some FAST for SharePoint labs one of the exercises had you edit a property called:
AllowNonCleanUpClaimsCacheForTestingOnly.
The production setting for this is “false”, which will make the system do cleanup of a claims cache.
But reading the name and what it actually does is very hard due to using both the words “Allow”, “Non” and “TestingOnly”. You have a positive word, a negative word and a specific case where the property is valid, all in the same name.
The proper name for this should in my opinion be:
CleanUpClaimsCache
where the production value should be “true”. With a comment to explain why you would set this to false, e.g. in a test scenario.
Sunday, February 27, 2011
FAST Search for SharePoint Query Logger
Saturday, February 26, 2011
Search Features vs. Security
Wednesday, February 16, 2011
Presentation at Norwegian SharePoint Community
I did a presentation at the Norwegian SharePoint Community (#NSPUG) meet-up yesterday. The presentation and code samples can be downloaded at http://cid-9ecc38025e460fc4.office.live.com/browse.aspx/.Public/%5ENNSPUG%202011-02-15
The presentation covers how you can utilize FS4SP as a data retrieval API outside the regular search page. The samples are simple, but shows the idea of expanding your toolbox with search as an API.
Friday, January 28, 2011
Using the SharePoint Search “Best Bet” feature to redirect to a landing page
Best Bets are recommended results. Best Bets can link to recommended Web sites, data stores, and documents. When a user includes a keyword or one of its synonyms in a query, the search results page features links to its associated Best Bets in a prominent position.This can look something like this for the search term “search” on my sample site:
Tuesday, January 25, 2011
How to: Frame a search center with little work
One way is to create a custom master page to replace v4.master for the search center, but this requires html knowledge and tinkering. In many cases it’s just as easy to go the configuration way.
Monday, January 17, 2011
How to “spy” the data in a custom pipeline extensibility stage with FS4SP
Fortunately for us, this stage still exists in FS4SP, and it might help you when testing and debugging your crawling.