Tuesday, December 30, 2014

Limit search results to “Documents”

image
An often asked question is to have Document Search, where the want is to list documents only - typically Office type formats like Word, Excel and PowerPoint. But the exact answer is not that white/black, and can differ from company to company.

One query you might consider executing to list documents only in SharePoint is IsDocument:true.

This query will return everything SharePoint deems a file or web page.. or not a list item (almost), returning Office documents, images, zip files… you name it, it will be part of the returned result set.

Which brings up the question: What is a document?

Monday, December 29, 2014

Solution to cannot search content in the Description field of a list

Update
See Site descriptions are now searchable!! (and more) for an update on the Description field.

Original
Got the scenario where someone had created a custom list in SharePoint and been so adventurous as to add a custom column to that list named Description. Talk about a bold move!

image
What happens in this scenario is that the column data is mapped to a crawled property named Office:6, and not ows_description which would be the natural property following the column –> crawled property naming convention. The Office category crawled properties are usually used for metadata coming from inside of Word, Excel and PowerPoint files, not list columns (See this old post from 2008), so I’m guessing an “edge case” bug :-)

Friday, December 26, 2014

The 100% way of automatically updating Result Types (on-prem only)

I have previously written The 100% way of getting your custom managed properties to show in your Display Templates, where after updating a display template with new managed properties you have to go to the Result Type settings page and click the “Update” link in the yellow alert shown. That’s what this post is about -  how to automate the manual click step.

image

Tuesday, December 23, 2014

How to enable Content Search Web Part Display Templates for Search Result Web Part

By default the out of the box control and display templates have been marked to work with their respective web parts. You can however re-use the templates across both web parts, as long as you ensure any script dependency is fixed. For example the CSWP has a reference to _layouts/15/search.cbs.js which must be added to the page in order for the Search Result Web Part to use the CSWP display templates.

The good thing is that the CSWP display templates are installed even though you don’t have the CSWP available. Here are the steps you need to perform to use the “Two Lines” CSWP display template with the Search Result Web Part.
  • Navigate to site settings
  • Under “Web Designer Galleries” go to “Master Pages”
  • Go to the “Display Templates” folder
  • Go to the “Content Web Parts” and you should see this listing
    image_thumb4

  • Edit properties on “Control_List.js”
  • Check “Search Results”
    image_thumb6

  • Click “Save”
  • Apply the same property change to “Item_TwoLines.js”
You have now enabled the display templates for use in the Search Result Web Part, and it’s time to set it up on a page.

Content Search Web Part vs. Search Results Web Part

Want the book? Go get it!

This is the bonus episode in my series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

If you have Enterprise license for SharePoint 2013 on-premises or you are using SharePoint Online Plan 2 (Office 365 E3/E4) then you have both web parts available. If you don’t have the E powers, you only have the Search Result Web Part, and this is what I will try to highlight. In what scenarios can you use the Search Result Web Part when you don’t have the license for CSWP?

Monday, December 22, 2014

Video Portal Troubleshooting

A new release without issues hard to achieve and Office 365 Video is no exception. I will explain a couple of typical scenarios which you might encounter.

  • A channel with that name already exists
  • I have renamed my channel, but the channel name is still listed with the old one on the channel listing page

Wednesday, December 17, 2014

S15E11: FAST Query Language (FQL) – The Unwanted Step Son

Want the book? Go get it!

This is the eleventh episode in my series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

In the first two episodes of the series I covered KQL and a couple of the new operators introduced with SharePoint 2013 which came from FQL - XRANK, NEAR and ONEAR.
In addition FQL has even more operators, some of them listed below:
  • starts-with
  • ends-width
  • count
  • andnot
  • range
There are also other operators which gives you great control over how a query is being sent over and parsed, and I recommend reading over the samples at FAST Query Language (FQL) syntax reference.

The post title plays at the fact that FQL is sort of hidden away, and no one really talks about it while the underlying search engine (and the refiner UI) in SharePoint (still) uses FQL.

This means KQL statements are translated to FQL before being executed. If this will hold true forever no one knows, but probably until that part of the search engine is being rewritten at some point – and this is good news as FQL has some tricks up it sleeve compared to the polished KQL counterpart.

Monday, December 15, 2014

Irregular gathering of crawled properties with SharePoint web crawls

To pick up data from <meta> tags when crawling web pages:

  • Make sure the crawled markup has line breaks after tags
  • Look in both Web and Document Parser crawled property categories for your crawled properties
  • Register the file extension crawled as the right mime type
  • Add the file extension crawled as the supported File Types

image

Thursday, December 11, 2014

How to trigger re-indexing of updated user profiles in SharePoint Online

You asked for it, and I deliver an early Xmas present for your pleasure and enjoyment!https://github.com/wobba/SPO-Trigger-Reindex

This script was made possible by the September 3rd release of the CSOM SDK which implements user profile write support. And DO NOT run it from a SharePoint Online Management Powershell, use a regular Powershell with no special cmdlets loaded. If not you get CSOM DLL hell.

Problem: You map an existing (or new) crawled property corresponding to a user profile property to a new managed property you want to use in search (phew). After the mapping, you wait and you wait and you wait (as you cannot start a full crawl yourself in SPO)… and you never get a value in your managed property. You curse some, you Google some, you post some questions in forums or Yammer. Then you go and edit your user profile in the SPO UI, wait some more.. and voila the value is there. You’re seriously baffled at the randomness and think to yourself

How the can I ask my 5,000 users to go edit their user profile to get that new property working in search??

(don’t worry - my script will help you out!)

Monday, December 8, 2014

The right robots.txt settings for allowing SharePoint to crawl your site

If you want you want to allow SharePoint 2010 or 2013 to crawl your web site add the following to your robots.txt file.

User-agent: MS Search 6.0 Robot
Disallow:

Even though the crawler sends Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot) as the user agent string, this is not what you should check against. Logical…. nah, but it is what it is.

Total cost to figure this out: 6h Sad smile

Reference: The SharePoint Server crawler ignored directives in Robots.txt

Friday, December 5, 2014

Tip for working with / troubleshooting search configurations in a production environment

I’m currently involved in a project where we have the typical DEV, TEST, STAGE, PRODUCTION way of deploying. Access wise you can typically view them like this:

DEV God rights!
TEST Demi-God rights!
STAGE Scream over the fence rights!
PRODUCTION Low life employee rights!

So, you’ve created your documentation and deployment packages, gone through all environments and finally handed them over to the PRODUCTION GOD. Hopefully all your meticulous steps have been followed, but alas, something is not working! SHOCKER! <insert scream here>

You have no way to examine configurations, as you can only search as yourself and have view rights.

In SharePoint 2010 you had to go to the SSA in order to view configuration, which was and still is very much hands-off unless you’re sitting in the same room or sharing a screen with the PRODUCTION GOD. In SharePoint 2013 Microsoft came up with this interesting concept that you could change the search settings not only on the SSA level, but on tenant, site collection and site level as well. From the PRODUCTION GOD’s point of view this is still evil as you need site owner rights to view search configurations, and you could potentially quick-fix stuff without them ever knowing – GOD FORBID ;-)

So I came up with this super duper awesome amazing solution which was approved by the powers that be.

Note: Our search configuration is deployed either to the SSA or the Site Collection level, as the search site is hosted at https://search.company.com. No site level configurations.

What I managed to pull off was for them to create a sub-site (any template really) with broken inheritance, where I was added as site owner. The site is also marked not to be indexed to avoid it showing up as a site hit. This effectively allows my regular account to go to that sub-site and view all inherited search configurations on schema, query rules, result sources and result types. If I tried to do a modification on this sub-site it wouldn’t matter at all, as I’m at a lower level than the production site, only messing up locally for that sub-site.

I am now able to troubleshoot and at check the configuration to make a more educated guess as to what the error might be before I engage with PRODUCTION GOD to fix matters.

First fix from this? I discovered some crawled and managed properties was not mapped causing a refiner to not show up – and it ended up being a step missing in the deployment documentation.

Wednesday, November 19, 2014

Black Market GUID’s

image

I’m currently sitting in the speaker room for SharePoint Connect 2014 in Amsterdam and somehow we got into talking about GUID’s. If you scour the interwebs you will find numerous sites offering to generate a unique GUID for you. There is even fraud sites where you can register and reserve GUID’s.

But what I did learn is that if you really want a truly unique GUID, one to keep and use all to yourself, one which will trump any other equal GUID held by someone else, then you have to go to the BLACK MARKET.

In every major city there are people dealing with black market GUID’s. They are truly unique and apparently of great quality. They have even survived the recent economic crises without so much as a dent or lost bit. I was lucky enough to buy one for my last euro’s second hand at the conference, and if everything goes my way I was told I will get the number for a local dealer in Oslo -  so as to scratch my daily needs for fresh GUID’s.

Unfortunately I cannot reveal the identity of the person selling it to me – but seems these black market dealers show up at larger conferences to hook more consumers of GUID’s.

Tuesday, November 18, 2014

Office Video is Live

Office Video just went live today November 11th, 2014. First they launched the Delve portal, now the Office 365 Video portal. Both driven by search. What’s next?

Getting-Started-1.jpg

I have had the fortune to test out the preview and have with my colleague Andreas written a guide to get you started planning this new awesome feature. Head over to Getting started with Office 365 Video to read all about it and also check out Office 365 Video Explained for some good insight into the platform.

Also check the official blog post for an introduction.

Tuesday, November 11, 2014

Tired of getting AllItems.aspx hits in your results?

If you append -contentclass:STS_List_* to your query template, they will all be excluded. In effect all list landing pages will be removed including:

  • AllItems.aspx
  • AllPages.aspx
  • AllPosts.aspx
  • AllCategories.aspx
  • Calendar.aspx
  • Overview.aspx

If you want to find all types excluded use the SharePoint 2013 Query Tool from Codeplex and issue the following query.

  • Row limit: 100 (or some high number)
  • Query Text: contentclass:STS_List_*
  • Collapse Spec: contentclass

The collapse specification ensures you only get one result per unique value.

Monday, November 10, 2014

How to: Make sure User Profile properties are free/full-text searchable

You get a requirement to add a new property called FavoriteColor to the user profile application, which you want searchable on the people search page.

First step is to create the property and ensure it will participate in search by checking the following two settings.


Once created, you fill it with values in your profiles, you wait for a crawl so that the crawled property People:FavoriteColor shows up, you create a new managed property named FavoriteColor, map your crawled property to it, fire off a full crawl for people, head over to people search page and type your favorite color.

Nothing shows!

However, if you type FavoriteColor:color, you get results.

I’ve gotten this question four or five times now, so thought I’d write it up.

The solution is as hard as it is easy. When you create a new managed property, it is by default added to the Default full-text index, not the PeopleIdx full-text index. The full text-index is what is used when you perform a free text query (as apposed to a property query). When querying for people you have to make sure your managed property is in the PeopleIdx.

On the managed property settings page, click the  Advanced Searchable Settings button, and pick PeopleIdx from the dropdown.

image

Next up perform a new full-crawl of the UPA, and it should all work! See https://www.techmikael.com/2014/12/how-to-trigger-re-indexing-of-user.html for how to re-index SPO user profiles, or take a look at the New-PnPUPABulkImportJob cmdlet.

You may also map the crawled property to the managed property ContentsHidden which is already set to be included in the PeopleIdx.

If you are curious what the FullTextIndex1/2/3 are, just stay away from them and don’t give it a second thought. They could in theory be useful, but when moving from FAST Search for SharePoint to 2013, some query logic was forgotten, so they are all but useless (known issue by the product team, and a ticket has been filed and archived). I might do a write up on that later – but not too many would be interested in it.

Wednesday, November 5, 2014

You should use exponential notation when writing XRANK queries


I have previously mentioned a problem using decimal numbers on XRANK statements in my Freshness Boost post. The problem is that the decimal separator used differs on your locale.
For example in Norwegian the character used is , (comma) while in English the character used is . (punctuation). The issue arises when a query rules is written using one locale, and being used on a query in a different locale.
Say I have a query rule with the following XRANK statement where I use . as the decimal separator
{searchterms} XRANK(cb=0.5) title:test
If a user has their SharePoint locale or browser settings set to Norwegian, then the query will fail, giving you an error message that search didn’t understand the query. The reason for the error is that the query framework will use .Net to parse the number with Double.Parse(), and this function is culture sensitive when parsing, and the culture used is based on the users locale settings.

The solution, as I have implemented in the Freshness Boost Generator, is to instead use exponential notation when writing the numbers. This means 0.5 is written as 5E-1 and 0.25 would be 25E-2. The number behind the E character is the number of decimal places to move to the left. This ensures that parsing the number always work, regardless of the locale used.
Rewriting the above query you end up with
{searchterms} XRANK(cb=5E-1) title:test
As a conclusion you should always use exponential notation when writing decimal numbers in an XRANK statement to ensure it will work across different locale settings.

Friday, October 31, 2014

Caution: If you have scripted your Search Service Application (eg. AutoSPInstaller) in SharePoint 2013 your search based pages could end up like this

Important!
If you have scripted your SSA, make sure the farm account has db_owner on your search db’s, or you may find yourself in a lot of trouble one day!

search-based-intranet

Wednesday, October 29, 2014

Retrieve Search Center URL from a SharePoint page

I have previously written about how you via REST can retrieve the Search Center URL for a particular SharePoint site.

If you are already on a page which has a search web part on it, you can get the same value from the page itself with the following command:

Srch.ScriptApplicationManager.get_current().states['searchCenterUrl']


To be clear; when a search web part exist on the page search Search.ClientControls.js is loaded and makes the Srch namespace available.

Tuesday, October 28, 2014

How to change the number of decimal places on a number column using CSOM

Got a challenge for a one liner by the strongest man in SharePoint (@tarjeieo) to do this using CSOM as with with SSOM you can set the DisplayFormat property of the field. Using CSOM, this property is not available.

A one liner would be ugly, but the way to accomplish this is to modify the SchemaXml of the field, so you could encapsulate the code below in a function.

Using XML manipulation I ended up with this.

ClientContext clientContext = new ClientContext("https://site/);
Web web = clientContext.Web;
Field field = web.Fields.GetByTitle("DummyNum"); // Field to modify
clientContext.Load(field);
clientContext.ExecuteQuery();
XmlDocument doc = new XmlDocument();
doc.LoadXml(field.SchemaXml);
XmlNode fieldNode = doc.SelectSingleNode("Field");
XmlAttribute decimalAttribute = fieldNode.Attributes["Decimals"];
if (decimalAttribute == null)
{
decimalAttribute = doc.CreateAttribute("Decimals");
fieldNode.Attributes.Append(decimalAttribute);
}
decimalAttribute.Value = "4";
field.SchemaXml = doc.OuterXml;
field.UpdateAndPushChanges(true);
clientContext.ExecuteQuery();

Monday, October 27, 2014

Caution when creating global query rules

If you create query rules which modify the query on your tenant or SSA level, then you are effectively killing off any such rules on the site collection level.

The below screen shot is taken running a query in the query rule tester. You see the first rule for blog’s is triggered. This rule has been set to stop, as it modifies the query and I want no more processing. You can see the stop flag actually works works, as the next rule is grayed out.

But then a rule from the SSA is showing in blue again, not honoring the Stop flag from the site collection. Ideally it should have been grayed out as well. The reason for this is that if you have more than one rule which modifies your query, then neither is executed (even though the test interface shows so).

image

In my case the SSA rule is adding an exclusion rule for all search results on Local SharePoint Results. The way to solve this is to create a custom result source on the SSA (or any other level where you need it), add the exclude filter here, and set this source as the default source instead.

Or if the results excluded shouldn’t be used anywhere, mark the site not to be indexed instead.

Tuesday, October 21, 2014

I'm doing an extra talk at SharePoint Connect 2014 - Social Refiners is the game!

Back in May I posted about SharePoint Connect 2014 letting me in to do a talk about Search Queries Explained. A vacant spot just opened up and I was asked if I could do one more, and I jumped at the chance.

I've decided to do a developer talk on how to create "Social" refiners on your search result pages.

The session will involve Display Templates and JavaScript with some FQL sprinkled in. I will show how to create my trusty old Department refiner for content and a refiner for content close to you, both based on your organization, who you follow and who you interact with from the Office Graph.

Hopefully you will pick up a trick or two and get some ideas for your next project - and if you are near sighted, sit up front as it's all code :)

Monday, October 20, 2014

Look up item based on the Item’s ID

If you kick off a search query you get some default properties with a document. One of these are DocId.

image

The managed property DocId is of type text and marked as searchable, and you would assume you a search like DocId=123456 will bring back a result back.

image

For some reason this does not work. The workaround however is easy.

There exists a managed property called WorkId which is also returned per result, but it’s not visible in the search schema. This property is of type integer with the same value as DocId and you can bring back one item with the following query:

WorkId:123456 or WorkId=123456

You can also do WorkId<>123456 or -WorkId:123456 if you want all items except one item in particular.

Alternatively you can map the crawled property ows__dlc_DocId to a managed property of your choice and use that instead, but no reason to do that since another solution exists.

Friday, October 17, 2014

Using search to redirect a user to the correct home site

image
I was recently tasked with the following challenge.

Given a two-level site structure of Region and Country, a user should be redirected to the corresponding site to where he/she works. This means if you work in Norway you should be redirected to http://intranet/About/Europe/Norway. If you work in Germany, which doesn’t have a country site of it’s own, you should be redirected to the region site at http://intranet/About/Europe.
  • About
    • Europe
      • Norway
      • Sweden
      • Denmark
    • APAC
      • China
      • India
      • Japan
    • Central Office
The URL structure of the sites replicate the taxonomy used for settings the values in Active Directory which is synchronized to the User Profile Application.

Thursday, October 16, 2014

Retrieving and working with Search Navigation entries using CSOM

I’m currently working on some provisioning tasks for a search page and we want to script up the search navigation using CSOM. This would be the Everything, People, and Social tabs you see below the search box.
The issue is that Web.Navigation does not have the SearchNav property which SPWeb.Navigation has. It only has QuickLaunch and TopNavigationBar.
But just because the good guys over at Microsoft haven’t gotten around to implement all the SSOM properties doesn’t mean we’re stuck. Using Reflector I figured out the node id for the search navigation was 0x410. Which yields the following code.
string siteUrl = "http://dev/sites/search";

ClientContext clientContext = new ClientContext(siteUrl);
Web web = clientContext.Web;

var nav = web.Navigation;

NavigationNode searchNav = nav.GetNodeById(1040);
NavigationNodeCollection nodeCollection = searchNav.Children;

clientContext.Load(nodeCollection);
clientContext.ExecuteQuery();

foreach (NavigationNode navigationNode in nodeCollection)
{
    Console.WriteLine(navigationNode.Title + " : " + navigationNode.Url);
}

Take a look at http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.client.navigationnodecollection(v=office.15).aspx for sample code on how to add more nodes to the collection.

Enjoy!

Wednesday, October 15, 2014

S15E10: Search Orchestration - Page composition in rule major

Want the book? Go get it!

This is episode ten in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

This post covers two main objectives. Firstly how to configure search for any page to make it manageable in the long run, and secondly how to configure search for that same page to make the page show as fast as possible to the end-user.

The main tool used to accomplish this is Result Block Routing in a Query Rule. If you haven’t read about it before, now is certainly the time.

image

How to: Quickly switch audio input in Lync 2013

image

When I move my computer around I tend to plug in and out different headsets, and Lync more often than not have the wrong one set as the default one. This results in that I a lot of times when on voice calls have to hang up, click the cogwheel icon, then Tools and Audio Device Settings, and pick the right audio device.

I’ve done this ever since Lync 2013 was released… up until today when I discovered a clever icon in the lower right corner of Lync. With two clicks I can now pick my Logitech headset – and hopefully remember to do it before a call starts.

Tuesday, October 7, 2014

Creating a Delve clone using Content Search Web Part

While adding support for Office Graph queries in the SharePoint 2013 Query Tool I had to figure out how to send the query in as POST and not just GET. This led me into the Properties property of a search query. Basically a key value store which Microsoft can use to extend the search API’s over time without breaking anything. And this is what the Graph queries use.

There has been some great posts recently from Richard Dizerega, Elio Struyf and Waldek Mastykarz about the Office Graph. Both Richard and Waldek have used the REST API, while Elio did some nifty JavaScript in order to run queries. What’s missing in Elio’s approach is to get Graph data on the initial loading of results, as the web part will first perform a regular query before you can graph it up and re-trigger it. But, as long as you manage to encode it just right, it’s all about editing a web part definition in your favorite text editor.

See demo

Note: Using CSWP with the Office Graph has to be done in SPO, and is not available on-premises (except perhaps using hybrid which I haven't tested)

Monday, October 6, 2014

Get a list of the current user’s recent documents

When navigating to OneDrive in SharePoint Online you get a nice list on the left showing your recent documents. The list show documents you have recently used, that being opened/viewed or stored. I suspect this is sort of overlapping with My work in Delve, but different API’s and results.

image

A recent discussion internally about bringing relevant content to you on your start page, made me dig into this. Put it as part of your intranet home page, or filter the site on any project site, to show your most recent files. It’s all about you!

So, how do you get this list of documents? By using a REST call to the end-point

https://<tenant>.sharepoint.com/_api/me/getrecentdocs

If you add the usual headers of application/json;odata=verbose, you get the data back as JSON. If not, you get it as XML. For me it returned 31 items, more than enough for a good listing.

Saturday, October 4, 2014

Office Graph support in the SharePoint 2013 Search Query Tool

I just compiled v2.2 of the SharePoint 2013 Search Query Tool which has one major new function: Support to query the Office Graph in Office365 using GQL (Graph Query Language). Many thanks to Barry Waldbaum @ Microsoft for starting the work on this feature, and the joint effort to bring it into the tool.

Thursday, October 2, 2014

Caution when limiting results for a site collection

One way to limit results to the current site collection in for example a result source or search web part is to use the following query template:

{searchTerms} SiteID:{SiteCollection.ID}

Turns out that there is a bug in SharePoint on this (present in both SPO and on-prem at the time of writing) where the ID returned for the SiteCollection is the ID of the root web, not the ID of the site collection itself. As per the MSDN documentation it should return the GUID of the site collection where the query was issued.

The workaround is to use the URL instead of the ID.

{searchTerms} SPSiteURL:{SiteCollection}

Don’t use managed properties with a dot (but the fix is easy)

KQL does not work with managed properties having a . in the name. (FQL however does).

This means if you try to query against a managed property named for example mAdcOW.CoolProp, you will get zero results back.

Two possible solutions, either rename your property or give it an alias. Using an alias is probably the better solution and allows you to give nice readable names to your properties. The alias feature is also nice when dealing with the re-usable RefinableXXYY properties, so as to name them properly.

In my case an alias was the way as the property was auto generated from a scenario like this:
  1. Site column with internal name mAdcOW.CoolDate
  2. Auto generated crawled property: ows_q_DATE_mAdcOW.CoolDate
  3. Auto generated managed property: mAdcOW.CoolDateOWSDATE
  4. Mikael manually added alias mAdcOWCoolDate
Hence, mAdcOW.CoolDateOWSDATE:Z does not work, while mAdcOWCoolDate:Z does work.

Using .'s in internal column names is pretty common for programmers to divide namespaces as part of the name, and usually they work out well. You could always change the naming of column names to avoid the propagation of dots to managed properties, but using an alias will still yield prettier names.

Friday, September 26, 2014

How To: Query the Office Graph using CSOM

I’m in the process of adding GQL support to the SharePoint 2013 Search Query Tool together with Barry Waldbaum (Microsoft), and while writing the POST support I found it easier to test it using CSOM.

image

Below is sample code with inline comments if you want to go the CSOM route instead of using REST to query the Office graph. The key is to use the Properties property of the KeywordQuery object.

The benefit of using CSOM over REST in my opinion is that you can pass in more than one query per call, and thus optimize network traffic and wire chatter.

Friday, September 19, 2014

How to remove certain items from a list/library in SharePoint, and keep all other items

The scenario is as follows. You have indexed a SharePoint farm and one of your business owners tells you that in his library he keeps multiple copies of a file, where the latest is marked as ApprovalStatus=1 with a custom column. And only those files should be surfaced in the search results.

So, how do you craft search a query? At first you might write something like:

ApprovalStatus=1

This will exclude all items which don’t have that value as well, so not a good idea.

Next you know the list has a custom list definition so you try something like:

ApprovalStatus=1 AND ContentClass=STS_ListItem_10300

Now you’re excluding all other content except that list. Even worse, but still explicit to what you want. The solution is to add an OR clause to include all the other stuff you want, which is everything.

((ApprovalStatus=1 AND ContentClass=STS_ListItem_10300) OR (ContentClass<>STS_ListItem_10300))

And we have arrived at the final query. It will include anything not from the 10300 list definition + items from the 10300 definition which is approved.

Conclusion

When you want to exclude certain content, it’s much easier to generate a rule saying what you want to include. In this case one part specifying the exact rule for what to show for a particular library, and one part saying everything else except that library.

Other possible solutions

  • Content enrichment web service which excludes content from that library and which is not approved
  • Don’t keep multiple files, but use SP versioning
  • Move non-approved documents to a library which is not being indexed

Thursday, September 11, 2014

Sending in arbitrary extra query parameters to your SharePoint Search Center

I touched on this subject previously in my post Appending query terms in SharePoint 2013/O365 without adding them to the search box, but I did not have a solution at that time. Now I have.
Topics covered
  • Extend Search.ClientControls.js to support arbitrary FQL statements
  • Add appended search queries to your page without them showing in the search box
As mentioned in Limiting search results by exact time in SharePoint 2013–and how to do FQL you can pass extra query information beside the search text itself by using the refinement filter parameter of a query. This parameter takes FQL by default which is very handy at times. So, the refinement filter property is not just for adding refiners, but can be used to add any FQL statement of your choice, that being an extra query term or XRANK statement.

Let that sink in and think about the possibilities. The good guys at FAST actually handed over the master key to search, probably without knowing it!

image

Monday, September 8, 2014

S15E09: Query Rules – Trigger happy circumstances (context)

Want the book? Go get it!

This is episode nine in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

image
This is one of my favorite areas of search, but also one of the harder ones to get right. The overall concept is as follows:

For a specific group of users you change the search experience based on that group.

In the sample image above I show a promoted result for new employees who happen to like StarWars. It’s a very simple concept, but a really powerful one, and it’s up to you to decide what your query rule should do with your result – promoted results, change sort order, change what is displayed etc..

Thursday, September 4, 2014

Debug crawled properties and creating a refiner for Content Source without modifying the existing ContentSource managed property

imageThat was a really long post title, but it explains what this post is about, so just look past the dryness of it.

The first part of the post explains how you can see what crawled properties are available for an item, and the second part is how you can use one of the RefinableStringXX managed properties as a refiner for content source. (Content sources is what you set up in your Search Service Application which points at what sources you want to index)

Say you have set up multiple content sources in SharePoint. One for SharePoint, a couple for file servers and maybe a web crawl or three of some web sites. Let’s assume all these sources have a mental footprint in the mind of your information works, and as such you want to show a refiner on the search page based on these content sources.

Out of the box SharePoint 2013 have a managed property named ContentSource which stores the name of the content source, but this property is not refinable, only queryable.
image

Monday, September 1, 2014

Importing Search Configurations on the SSA

imageYou might have noticed the settings for site collections have options to export and import search configuration files, which is very nice indeed. The Search Service Application however, does not have these options. What is available is the use of CSOM to export the SSA configuration via the SearchConfigurationPortability class. If you try to import a configuration file to the SSA you will simply get a nice error, explained at the documentation page for SearchConfigurationPortability with this interesting quote: ….however, you cannot import customized search configuration settings to a Search service application (SSA).

 Note: This does not apply to SharePoint Online where you can import/export search configuration from the UI for your tenant.

So, you have export, but no import!

Thursday, August 28, 2014

PDF previews enabled in Office365 for search and document libraries

image

With the March 2013 PU SharePoint 2013 on-premises got this functionality as Office Web Apps added support to open PDF documents. Not a big surprise as Word itself can open PDF documents.

For on-premises you could for search results easily add a Result Type to allow for PDF previews, while for document libraries the best option is Wictor Wiléns farm solution which patches up, or my more ad-hoc solution as blogged at http://techmikael.blogspot.com/2013/06/enabling-pdf-previews-in-your-lists.html

Long ago this was also available in SharePoint Online, but it was removed at some point for unknown reasons – meaning they turned off the PDF support for the online WAC server. Good news is that they are back!

The updates themselves

For search it is the Item_PDF_HoverPael.html file which has been updated - _catalogs/masterpage/Display Templates/Search/Item_PDF_HoverPanel.html. Hopefully this update will find it’s way on-premises as well, or simply copy it from an online tenant locally.

For document libraries the filepreview.js file has been patched to support .pdf, same as Wictor and I have blogged about.

Tuesday, August 26, 2014

Limiting on types of file in SharePoint

There are two managed properties you might consider for limiting or refining on the types of files indexed in SharePoint. One is FileExtension, the other is FileType. The main difference is that FileType is based on the content inside the document, not the extension, while FileExtension merely strips of the file extension of the file name.

This means that if you have a text file named test.txt and rename it to text.xlsx, on the FileType refiner it will be sorted under txt, not xlsx.

My personal preference and SharePoint’s default behavior is to use FileType.

When it comes to Display Templates, SharePoint will by default use FileExtension. This means if you rename test.xlsx to test.docx, it will use an Icon for Word, but show a refiner for Excel – unless you override and change the behavior in your Display Templates that is.

image

Thursday, August 21, 2014

How To: Search up items which don’t have a value set

imageOne of the harder things in search is to search for something which don’t have a value set, and it sort of goes against how a search engine works. If an item is missing a value on a property, then this property is not set in the search index. It will be non-existent.

Using SQL you can use operators such as IsNull to return items which are missing a value, but using SharePoint search you don’t have this operator.
A common work around is to provide some dummy/default value, and you can filter on this one instead. But often this is not an option.

Tuesday, August 19, 2014

“Optimize for frequent use” setting in a Result Type

image

You have probably seen the check box for Optimize for frequent use if you have ever created a Result Type in SharePoint. But, what does this mean?

Saturday, August 9, 2014

S15E08–Query Rules – I am Trigger Happy (intent)

Want the book? Go get it!

This is the eight episode in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

In this fourth episode around query rules, I am going to focus on capturing the intent of the user. What is the person searching really looking for? If you look at a random query log you can, more often than not, easily pick out queries which you could call “Q and A queries”. By “Q and A query” I mean a query which is meant to bring back an answer which you can take action on right away.

Wednesday, July 23, 2014

Gotcha on when a Query Rule Doesn’t Trigger

A small heads up if you are creating query rules, and you notice that they won’t trigger. If you include a KQL operator or wildcard as part of your query, the rule won’t trigger.

Consider the screenshots below where the first one shows the wiki block, and the rest don’t. And it doesn’t matter which triggering mechanism you choose, they won’t trigger.

Reading closely on the setup page this is expected behavior:

image

You can also test this using the SharePoint 2013 Query Tool or the Query Tool test feature on the Manage Query Rule settings page in SharePoint.

image

image

image

image

Tuesday, July 22, 2014

Custom Search Result Page Layout in SharePoint

You’re messing with the search result page and figure you need a new zone on the page for some cool web parts.

Being an avid SharePoint Designer fan as I am, you open up the site and copy _catalogs/masterpage/SearchResults.aspx to _catalogs/masterpage/SearchResults_custom.aspx. Then you add a new web part zone to the bottom of the page and save the layout.

Next you make the layout available on your site (Bing it) and create a new page based on the layout.

Boom! – Code blocks are not allowed

image

Turns out the fix is pretty easy as the file actually has code blocks in it.

image

Simply remove the <% %> parts which does conditional checks for the Ribbon being hidden if the user is in Browser mode (not Design, Catalog or Edit mode)

You can probably implement the check yourself in JSOM if needed, or simply show/hide the ribbon in all modes.

So, how come the default layout works with code blocks? Well.. SharePoint does whatever it wants with the default pages and can configure SharePoint to work as it pleases. Being an outsider you don’t have those powers.

Monday, July 21, 2014

Better Best Bets with Lists

imageBest Bets or Promoted results is a great way to give end-users the answer to their search query without the additional click to open a page or document to locate the precise information they were looking for. Bing and Google already have many examples of this if you for example look for weather information or currency conversion.
Head over to the Microsoft MVP Award Program Blog, and read more about how you can improve on SharePoint’s built-in solution!

Wednesday, July 9, 2014

July 2014 CU for SharePoint 2013 released–with a search feature long lost from 2010

You may now install custom PDF iFilters again, like the one from Foxit or Adobe! In SharePoint 2010 you could install any iFilter you wanted for text extraction and the most common to install was for PDF documents. With SharePoint 2013 this was replaced by an internal file handler instead, which could not be overridden.

With the July 2014 CU for SharePoint 2013, you can yet again install your custom PDF iFilter, or override any built-in handler to use the iFilter of your choice.

Friday, July 4, 2014

Mikael’s best practice for managing Managed Properties in SharePoint 2013/SPO–and how to deal with dates

I’ve had this on my mind for a while, but  Waldek Mastykarz post “Inconvenient Managed Properties and getting a list of all custom Managed Properties in SharePoint 2013” triggered me to share my thoughts.

To sum up Waldek’s posts, he shows a neat way to use PowerShell on-premises to filter out all non-system defined managed properties. The disadvantage is that the list will also include all managed properties auto generated from site columns. And you cannot use it for SharePoint Online.

One of the lacking points with the auto generated managed properties, is that they are of type text, even for date time values. If you want to filter on date ranges you need to use a managed property of type Date and Time which is sortable, and not a text property. As Waldek points out, the re-usable properties of RefinableDateXX are lacking as they return a string object and not a date object. They work for filtering, but in a Display Template they suck. This argument is used by Waldek to point out that you end up creating custom managed properties to deal with for example dates – and the argument is quite valid and not a bad one. Because Waldek is a pretty darn smart guy if you haven't noticed and his blog is certainly one to follow.

image

What makes a SharePoint column searchable?

By searchable I mean a user searches for a term, and get a search result if that term exists in a SharePoint column (without specifying a property query). Basically typing keywords in a search box.

By default when you create a SharePoint column on a list, it will generate a crawled property which is marked with Include in full-text index. This checkmark ensures you will get a hit for terms in this column. The naming of the crawled property is ows_internalColumnName.

If you look at managed properties, they have a corresponding property called Searchable. This means if your crawled property is mapped to a managed property marked as searchable, you will get recall on the column. However, if the managed property is not marked as searchable, even though the crawled property is marked with “Include in full-text” index, it will NOT be searchable.

To put it all in a table.
Crawled Property Managed Property
Included in
Full text-index
Not included in
Full text-index
Searchable Not
Searchable
Column
Searchable
x
x
x x
x x
x x
x x

Note

If a crawled property is mapped to two managed properties, where one of them is searchable, then the value will be searchable.

If a column is hidden or marked via column properties as "NoCrawl", then it will not be searchable.

If the crawled property is from a number/currencly column, you need to map it to a searchable managed property in order to get full text index recall.

Summary

In order to make a column in SharePoint not searchable, you either have to uncheck the option on the corresponding crawled property to include it in the full-text index, or map it to a managed property which is not marked as searchable.

Check out Benjamin Niaulin's post Understand SharePoint Crawled and Managed properties for search for more information on crawled and managed properties.

Friday, June 27, 2014

S15E07: Query Rules – Get things in order

Want the book? Go get it!

This is the seventh episode in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

In the “Rewrite and paraphrase” post I discussed the concepts of recall and precision which has to do with bringing back relevant items for a search query. When you have your golden set of items, the next step is to sort them in the best order possible to ensure the most relevant items are in your top hits or first page of results.

Tip: Increase the default number of items to display to 25 or 50. Ten items is not a lot and people are getting more and more used to scrolling these days.

There are four ways to sort a result in SharePoint:
  1. Sort the results based on a ranking profile
    • You may also influence ranking by using the XRANK keyword (dynamic ordering)
  2. Sort the results based on a managed property ascending or descending
  3. Sort the results based on a sorting formula
  4. Sort the results randomly
    • This is accommodated using a special sorting formula

Monday, June 16, 2014

SharePoint Saturday Oslo–In Review

It’s now been a little over two weeks since the second SharePoint Saturday Oslo was held May 31st, 2014. And I have to say it was a blast! Everyone I talked to said they had a good time and that they enjoyed the event. I also want to thank the sponsors for making this possible, in particular K2, Metalogix and Steria. Couldn’t have done it without you!

image

 

Friday, June 13, 2014

S15E06: Query Rules–Rewrite and paraphrase

Want the book? Go get it!

This is episode six in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

image

Two key concepts in search are recall and  precision. Recall refers to the number of results a search engine returns for a specific query, and precision refers
 
to how precisely the results match the search user’s intent. You must make a compromise between recall and precision. If recall is too large, you essentially flood the result set with noise, and precision suffers. On the other hand, when recall is too small, you run the risk of missing useful results, so the item(s) the user is looking for may not appear in the result set; again, precision suffers.

And this is where Query Rules play a part in SharePoint. By looking at the queries coming in, you have the option to change them to better match your content, and thus improve on recall and precision.

Thursday, May 29, 2014

Display Templates from my SPC14 and ESPC14 talks

At SPC14 and ESPC14 I used some custom display templates where I slightly changed the default search center UI. Included is:

  • counts to the vertical tabs
  • icons for file formats
  • department refiner for content
  • styled the best bets
  • context wire-up for User Segments triggering in query rules
  • blog post template
  • document link template

The templates work equally well for SharePoint Online as for on-premises.

image

You can find the templates over at the SPCSR community project at github if you navigate to DisplayTemplates / Search Display Templates / mAdcOW Search Center Templates.

For those interested, check out my ESPC14 slides.

And see David Hollembaek’s and my SPC14 presentation and slide deck over at http://channel9.msdn.com/Events/SharePoint-Conference/2014/SPC382

Wednesday, May 28, 2014

Themes in SharePoint–not 100%

image

image

If you look at the above images, the ribbon still has some blue lines. But the right part of the suit bar is fixed.

S15E05: Query Rules – The Basics

Want the book? Go get it!

This is the fifth episode in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

I want to start out by saying that Query Rules Rule!! And in a big way!

If you are coming from SharePoint 2010 and used keywords for synonyms and best bets in search, then Query Rules is the brand new revved up engine which took it’s place. Think an old Toyota with a 1.4l engine, being replaced by a Tesla Model S (or The Expendables without steroids).

So what is a query rule exactly? When a search query is being executed, one or more query rules can based on specified conditions intercept the query and take actions on it.
The available actions are:
  1. Add promoted results (Best Bets in 2010)
  2. Add one or more result blocks, which are additional groups of results or side results to the original result
  3. Re-write the query
  4. Change the sorting or ranking of the results
image

Tuesday, May 27, 2014

Speaking at SharePoint Connect 2014

At the upcoming SharePoint Connect 2014 in Amsterdam (November 18th-19th) I will do a deep dive into the fun stuff surrounding search – Query Rules, Query Variables and Search Orchestration.

278x200-Mikael-Svenson

The session will be built around my SharePoint Search Queries Explained series and I will present some good patterns and practises around wiring up content search. When you leave you should have some ideas about how you can optimize and utilize the capabilities in SharePoint 2013 and SharePoint Online.

The line-up for the conference is pretty good and I’m proud to have the only IT Pro/Dev session. Home in on the single green square on the agenda and you’ve got my slot!

Register now and hope to see you there! Use the discount code SA244 on the last page of registration and get a 10% discount!

Monday, May 26, 2014

S15E04 Result Sources–Scope it out!

Want the book? Go get it!

This is the fourth episode in the series “SharePoint Search Queries Explained - The Series”. See the intro post for links to all episodes.

Result sources are one of the key concepts when it comes to SharePoint search, and they are key to descending where and how a search query is to be executed. If you have previously worked with SharePoint 2010 search, start thinking about scopes and you should be on the right track. But result sources are so much more.

Wednesday, May 21, 2014

Subtle changes in the suite bar in SharePoint Online

Here’s what it looks like in the old version (v16.0.0.2811):
image
And here’s the new one (v16.0.0.2902):
image
As you can see the white has gone gray, and the part to the right has gone from gray to light blue. All in all making the suite bar a bit more subtle.

But this time the CSS class names were not changed - phew!

[Update as per Oliver’s comment]

Seems there is a bug with the new theme. If I try a composed look it looks like this.

image

The top right does not change.. .we have a bug “again” people :)