Retirement of custom default result sources in Microsoft Search for modern search experiences

In November 2021 I posted about bookmarks being the successor feature of promoted results for organizational scoped searches in Microsoft Search, which was the first step to modernize the Microsoft Search stack and remove dependencies on classic SharePoint search features and API’s.

The next step is now under way, as announced by MC526131 - Retirement of custom default result sources in Microsoft Search for modern search experiences.

 

For most customers the change, which will start April 10th 2023, should have no impact. The way KQL rewrite works for a default result source was never intended for modern search experiences as any change was applied to all verticals showing SharePoint and OneDrive content. The ability for an admin to edit and add KQL per vertical in the modern experience is a better and more accurate feature – succeeding the result source feature which doesn’t really work well in modern search experiences in SharePoint. See https://learn.microsoft.com/en-us/microsoftsearch/manage-verticals#keyword-query-language-kql for more information on vertical management.

I want to be crystal clear that nothing happens to classic search experiences nor to experiences which are powered by the SharePoint Search API. Everything keeps on working as before – it’s just Microsoft Search in out-of-the-box modern experiences which retires reading the setup.

I also want to point out that this does not effect query rule triggered promoted results on SharePoint sites or SharePoint hub-sites, as the modern experience will show a promoted result for these scopes regardless of the result source they may have been targeted towards.

And as a last note, this only applies to environment where search vertical administration is rolled out.

That’s it and you can likely ignore this post as it should not affect you

Surface Dynamics 365 Data in Microsoft Search

If you’re a user of Dynamics 365 did you know that it’s now possible to surface search results from Dynamics 365 in Microsoft Search?

Head over to https://docs.microsoft.com/en-us/microsoftsearch/manage-dynamics365, or direct your admin to follow the directions on how to enable the experience.

Quick steps:

• Navigate to https://admin.microsoft.com/Adminportal/Home?#/MicrosoftSearch/connectors
• Click “add a new app or service”
  
• Pick D365 and follow the setup wizard
  

Sensible defaults – avoiding error on first navigation to the Files tab in a Teams channel

I won’t take credit for the solution myself, as this was shared to me by the awesome Remi Blom-Ohlsen, who is too timid to share great tips with the world himself :)

We’ve all been there, and if you create solutions for customers involving Teams you are bound to have encountered the issue. After the team is created and you immediately navigate to the Files tab in a channel, you get an error message.

Navigate to the Posts tab and back, and all is working.

When creating Teams using out of the box user interfaces you still have to live with this. If you however use a custom ordering and provisioning solution you can via code fix this nuance for your users :)

If you are patient and wait a tiny bit, the folder will be created for you. For those who are impatient and want to ensure everything is the best way possible for end user, they can continue to read.

If your provisioning solution is using C# you can ensure the channel folder is creating using code like this. And the crux here is the FilesFolder property on the channel object.

public static async Task<DriveItem> InitTeamDrive(string GroupId, GraphServiceClient GraphClient = null)
{
	var graphClient = <get your client here>;
	var channels = await graphClient.Teams[GroupId].Channels.Request().GetAsync();

	foreach (var channel in channels)
	{
		if (channel.DisplayName == "General")
		{
			var drive = await graphClient.Teams[GroupId].Channels[channel.Id].FilesFolder.Request().GetAsync();
			return drive;
		}
	}
	return null;
}

The similar direct graph call would be.

GET https://graph.microsoft.com/v1.0/teams/<group id>/channels/<channel id>/filesFolder

Phase 1 of Bookmarks to take over for promoted results in Microsoft Search

As announced in the M365 message center post MC293777 Microsoft Search will over the course of November to December of this year execute phase one to deprecate the support of classic query rule based promoted results / best bets, and replace it fully with Bookmarks and Q&A answers.

Message center announcement: In order to improve the Microsoft Search experience, promoted results will no longer be supported in Microsoft Search for Organization level scoped searches and will be replaced by the Bookmarks and Q&A features.

Updated script to re-index user profiles for search

This has been on my mind for years and I finally got the time to create a new version of my PowerShell script to re-index user profiles for the scenario where you map a new UPA property to be used in search.

You can pick up reindex-users-v2.ps1 from https://github.com/wobba/SPO-Trigger-Reindex. The script allows for quicker updates especially for tenants with a lot of profiles as it uses the UPA bulk import API instead of iterating the UPA.

See https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/bulk-user-profile-update-api-for-sharepoint-online and https://docs.microsoft.com/en-us/previous-versions/office/sharepoint-csom/mt642955(v=office.15) for documentation.

Cover image by Markus Winkler @ Unsplash

Site descriptions are now searchable!! (and more)

“What????!” you might say, hasn’t it always been like that? And the answer is that ever since SharePoint 2013 it hasn’t. Because if it had, I would not be writing this – just saying.

As a bonus if you create a column in a library named Description, the content of this property is now also searchable. See my old post https://www.techmikael.com/2014/12/solution-to-cannot-search-content-in.html about this issue.

That’s correct people, you get a 2 for 1 special as Microsoft Search has rectified a long time nuance. And I’ll take the credit for getting this fixed (with some awesome technical help from Himanshu on the Stream team) - I can be humble another time

Note: The change applies to updated content, not content already in the index. For active sites it should fix itself, so no action needed on your part.

Technical explanation

For the site description (or the description of an Office 365 group or a Team) the crawled property storing the description value, ows_SiteDescription, was mapped to two managed properties:

• Description
• SiteDescription

Neither which were marked as searchable, effectively removing the ability to get a match on normal user queries from a search box. We have now made a search schema change to map ows_SiteDescription to the managed property Contents in addition to the two existing ones, ensuring full-text search.

Same goes for a SharePoint column/field named Description. Except this is a weird one as the crawled property for this column is Office:6, not ows_Description. To be on the safe side we have mapped both Office:6 and ows_Description to Contents as well.

Need a refresh on what makes content free-text searchable in SharePoint? Head over to https://www.techmikael.com/2014/07/what-makes-sharepoint-column-searchable.html.

Cover image by Alexander Krivitskiy@unsplash

Microsoft Graph: Encoding and decoding the drive id

If you have worked with the Microsoft Graph API and SharePoint items you might have encountered URL’s which include a drive id. Where you in SharePoint typically work with URL’s for a site and the document library, a drive id is an encoded representation of a document library location.

The following Graph REST request will list all items in a specific document library:

GET https://graph.microsoft.com/v1.0/drives/b!VvpCx03990mC5Lb5YxH0SUA9TgZHvEZImra6PMjrvbx85KUwT1BMTbhen6I6ffXL/root/children

Fetching the item for a file you use this signature to retrieve the file itself where item id is part of the listing: GET /drives/{drive-id}/items/{item-id}

With some reverse engineering you easily can figure out the drive-id is a prefixed base64 encoded string composed of the site id, web id and list id for a particular library.

Using PowerShell here’s a few lines converting the drive id string to the correct guid’s and back:

$driveId = "b!VvpCx03990mC5Lb5YxH0SUA9TgZHvEZImra6PMjrvbx85KUwT1BMTbhen6I6ffXL"   
$encodedDriveId = $driveId.Substring(2).Replace('_','/').Replace('-','+')    
$bytes = [Convert]::FromBase64String($encodedDriveId)
$siteIdBytes = [byte[]]$bytes[0..15]
$siteIdGuid = New-Object Guid @(,$siteIdBytes) #site id

$webIdBytes = [byte[]]$bytes[16..31]
$webIdGuid = New-Object Guid @(,$webIdBytes) #web id

$listIdBytes = [byte[]]$bytes[32..47]
$listIdGuid = New-Object Guid @(,$listIdBytes) #list/library id

$bytes = $siteIdGuid.ToByteArray() + $webIdGuid.ToByteArray() + $listIdGuid.ToByteArray()
$driveId = "b!" + [Convert]::ToBase64String($bytes)

The item-id part is a bit more tricky and looks to be some sort of base32 encoding of a SharePoint item’s unique id. As I haven’t figured out the mechanics you can still access the item by route of the list item unique id:

GET https://graph.microsoft.com/v1.0/drives/{drive-id}/list/items/{unique-id}/driveItem

Example: https://graph.microsoft.com/v1.0/drives/b!VvpCx03990mC5Lb5YxH0SUA9TgZHvEZImra6PMjrvbx85KUwT1BMTbhen6I6ffXL/list/items/7D0E814C-0798-4966-A20C-6DE4F27F2027/driveItem

Of course, if you already have the site id, web id, list id and unique item id, you can access a drive item using the following syntax as mentioned in my file preview post.

GET https://graph.microsoft.com/v1.0/sites/{site-id},{web-id}/lists/{list-id}/items/{item- id}/driveItem

Photo by https://unsplash.com/@maurosbicego

Mapping or clearing crawled property to managed property mappings using PowerShell (or code)

Mapping a crawled property to a managed property in SharePoint Online is usually done via the user interface. A somewhat cumbersome task as you first have to create content with a value which will ensure the creation of a crawled property matching your SharePoint field (or column if you want). Once the crawled property is available you can map it to a managed property.

Lately there has been issues where crawled properties are not listed in the admin UI due to a bug, and this blocks tenants from creating search solutions dependent on schema mappings. There are some workarounds available which you can check out at https://platinumdogs.me/2020/11/13/sharepoint-online-search-crawled-properties-not-created-not-showing-or-not-available/.

Instead of these workaround I figured I’d show how you can use search configuration XML files to accomplish the same task – a pattern I used many times back in the day when working on SharePoint 2013 on-premises and had to move search configurations between environments. These files together with the PnP cmdlet Set-PnPSearchConfiguration is a time saver. You can also use

Get-PnPSearchConfiguration -Scope Site -OutputFormat ManagedPropertyMappings

to see any mappings at a particular site.

The first bonus of my solution is that you can map a crawled property which don’t yet exists, as it will be created for you. No need to create content before you start mapping.

The other bonus is that you can also clear out an existing mapping, reverting it to the default behavior. Particularly handy if you at a site scope override a global mapping, and later want to revert. Something you simply cannot do in the admin interface today. 

Note: The best way to create these files is to use a new site collection in either SharePoint Online or SharePoint 2013/2016/2019 where you are able to do the mappings. Manually create the needed mappings and export the search configuration file to be used in another environment (site, or tenant).

The script currently works with ows_<fieldName> crawled properties. Which means you need to figure out the name of the crawled property manually. (ows_ + internal name of the SharePoint Field). It will also only work with the reusable managed properties as we know the id’s of these beforehand. In order to make it work for other crawled properties the script must be expanded to support more property set id’s. Focusing on the set 00130329-0000-0130-c000-000000131346 should however cover most scenarios.

Head over to https://github.com/wobba/SearchConfiguration and fetch the files:

• MapCrawledPropertyToManagedProperty.ps1
• SearchMappingTemplate.xml
• SearchMappingReset.xml

Check the top of the ps1 for usage. Happy property mapping!

Cover photo by Markus Winkler @ Unsplash

Search and modern multilingual pages in SharePoint Online

The ability to create SharePoint pages in multiple languages was/is a popular feature for classic SharePoint publishing pages. If you work in an organization where your employees use multiple languages, for example English and Norwegian, creating side by side pages in native languages is beneficial both from a communication point of view and also helps to tailor the information related to countries/regions based on language.

Back in SharePoint 2013 the functionality was called variations. This time around it’s more aptly named the “multilingual feature”. You can read the feature announcement from June 2020 at the Microsoft Tech Community.

While variations were an option for sites using the classic SharePoint publishing feature, the ability to create multilingual pages is currently enabled by default for Communication sites only.

If you want to enable multilingual page support on a Team site you can do so by activating a hidden feature called SitePagePublishing using for example PnP PowerShell as a site owner.

Connect-PnPOnline https://contoso.sharepoint.com/sites/mysite
Enable-PnPFeature -Identity f39dad74-ea79-46ef-9ef7-fe2370754f6f

If you want to take advantage of the ability to author multilingual pages, head over to Create multilingual communication sites, pages, and news which describes in detail how to turn on the capability as well as how to author the pages.

Details needed for search

What happens behind the scenes when you create multilingual pages? For one, each language copy is stored in its separate folder. In my case English is the primary site language, and I opted to have Norwegian as a secondary language.

Two pages in the original language.

One of the pages is translated.

Secondly, to connect and keep track of the original vs. the translated pages there are some internal properties in SharePoint per item keeping track of this.

SharePoint Column	Search Managed Property	Description
_SPTranslatedLanguages	SPTranslatedLanguages	Source page stores a list of the language of all translated languages. E.g. nb-no.
_SPTranslationLanguage	SPTranslationLanguage	Translated page stores the language of the page. E.g.: nb-no
_SPTranslationSourceItemId	SPTranslationSourceItemId	Translates page stores a reference to the original page. E.g: 1f50aa8b-d31e-415a-bc4a-4ced38f5cdc4
_SPIsTranslation	SPIsTranslation	The translated page stores the boolean value true to indicate it’s a translation and not the original. The original article has no value.

The desired behavior when searching is to return a page in the users’ display language if it exists. If a translation does not exist, fall back to the original language – which follows the default language of a site set at site creation.

The examples below explain the logic used by Microsoft Search, and you can replicate this yourself as well in custom solutions.

To return English pages returned the query would be:

(SPTranslationLanguage:en-us OR (NOT SPTranslatedLanguages:en-us AND NOT SPIsTranslation:true))

Both pages returned in the original language - English.

To return Norwegian pages the query would be:

(SPTranslationLanguage:nb-no OR (NOT SPTranslatedLanguages:nb-no AND NOT SPIsTranslation:true))

One translated page and one non-translated returned.

If you are using the PnP Modern Search web parts you can use the query variable {PageContext.currentUICultureName} to substitute the correct display language into the query.

Easy as pie :)

PS! For the KQL savvy, you can not replace the NOT operator with the – (minus) operator for boolean values – for some weird reason that fails.

Cover photo by Mat Reding @ Unsplash

Microsoft Search vs. PnP Modern Search – a clash of titans!

Sorry for the clickbait title. I have been meaning to write about how I think about the out-of-the-box Microsoft Search vs. the PnP Modern Search web parts for a while, so here goes.

Disclaimer: I have been working with search for 20 years, enterprise search for 15 years and SharePoint and search for almost just as long. The opinions stated are my own and they are based on my experience with search in FAST ESP, SharePoint and Office 365 over the years and are not influenced by my role as a program manager on Microsoft Search. No money or breaking of fingers were involved in writing this post :)

A little history

With SharePoint 2013, Microsoft shipped many ways to customize how search worked. Both from a technical and user experience perspective. The more important one being the ability to use arbitrary HTML, CSS, and JavaScript in custom display templates with the Search Result web part and the Content by Search web part (CSWP).

Fast forward to SharePoint Online and the release of modern SharePoint pages late 2016. Modern pages came with a new set of web parts and lacked the super-flexible CSWP. Modern pages did introduce the Highlighted Content web part (HCWP), positioned as “...the newer, simplified version of the Content Search web part.”

When customer transitioned from classic to modern a gap had been formed between the CSWP and the HCWP which needed filling. And this is where the awesome sharing in the SharePoint community comes in, put into a sharable system with Microsoft 365 Patterns and Practices (PnP).

The first version of the PnP Modern Search web parts (MSWP) were added as a SharePoint Framework sample by Franck Cornu back in October of 2017. My first contribution to the sample was made in January of 2018 to include additions needed for a customer project I had at the time.

Back then MSWP was one web part; the result web part which included an optional built-in refinement panel. The search box web part got introduced late February of 2018, with the vertical web part following later. My main reason for looking at the sample web part in the first place was the need of a roll-up web part for a project. The Highlighted Content web part (HCWP) lacked the needed templating required – and still does, and why bother develop something if it already exists? Much easier to add to a maintained code base compared to owning it completely on my own #lazyprogrammer.

Image by Zhang Kenny @ Unsplash.

And this is where I believe the PnP Modern Search web parts has its sweet spot – bridge the gap between the classic Content by Search web part and the new Highlighted Content web part. This was my opinion back in 2018, and it still is in 2020. I actually formed the opinion back in 2016. That is 4 years and counting.