Mapping or clearing crawled property to managed property mappings using PowerShell (or code)

Mapping a crawled property to a managed property in SharePoint Online is usually done via the user interface. A somewhat cumbersome task as you first have to create content with a value which will ensure the creation of a crawled property matching your SharePoint field (or column if you want). Once the crawled property is available you can map it to a managed property.

Lately there has been issues where crawled properties are not listed in the admin UI due to a bug, and this blocks tenants from creating search solutions dependent on schema mappings. There are some workarounds available which you can check out at https://platinumdogs.me/2020/11/13/sharepoint-online-search-crawled-properties-not-created-not-showing-or-not-available/.

Instead of these workaround I figured I’d show how you can use search configuration XML files to accomplish the same task – a pattern I used many times back in the day when working on SharePoint 2013 on-premises and had to move search configurations between environments. These files together with the PnP cmdlet Set-PnPSearchConfiguration is a time saver. You can also use

Get-PnPSearchConfiguration -Scope Site -OutputFormat ManagedPropertyMappings

to see any mappings at a particular site.

The first bonus of my solution is that you can map a crawled property which don’t yet exists, as it will be created for you. No need to create content before you start mapping.

The other bonus is that you can also clear out an existing mapping, reverting it to the default behavior. Particularly handy if you at a site scope override a global mapping, and later want to revert. Something you simply cannot do in the admin interface today. 

Note: The best way to create these files is to use a new site collection in either SharePoint Online or SharePoint 2013/2016/2019 where you are able to do the mappings. Manually create the needed mappings and export the search configuration file to be used in another environment (site, or tenant).

The script currently works with ows_<fieldName> crawled properties. Which means you need to figure out the name of the crawled property manually. (ows_ + internal name of the SharePoint Field). It will also only work with the reusable managed properties as we know the id’s of these beforehand. In order to make it work for other crawled properties the script must be expanded to support more property set id’s. Focusing on the set 00130329-0000-0130-c000-000000131346 should however cover most scenarios.

Head over to https://github.com/wobba/SearchConfiguration and fetch the files:

• MapCrawledPropertyToManagedProperty.ps1
• SearchMappingTemplate.xml
• SearchMappingReset.xml

Check the top of the ps1 for usage. Happy property mapping!

Cover photo by Markus Winkler @ Unsplash

Search and modern multilingual pages in SharePoint Online

The ability to create SharePoint pages in multiple languages was/is a popular feature for classic SharePoint publishing pages. If you work in an organization where your employees use multiple languages, for example English and Norwegian, creating side by side pages in native languages is beneficial both from a communication point of view and also helps to tailor the information related to countries/regions based on language.

Back in SharePoint 2013 the functionality was called variations. This time around it’s more aptly named the “multilingual feature”. You can read the feature announcement from June 2020 at the Microsoft Tech Community.

While variations were an option for sites using the classic SharePoint publishing feature, the ability to create multilingual pages is currently enabled by default for Communication sites only.

If you want to enable multilingual page support on a Team site you can do so by activating a hidden feature called SitePagePublishing using for example PnP PowerShell as a site owner.

Connect-PnPOnline https://contoso.sharepoint.com/sites/mysite
Enable-PnPFeature -Identity f39dad74-ea79-46ef-9ef7-fe2370754f6f

If you want to take advantage of the ability to author multilingual pages, head over to Create multilingual communication sites, pages, and news which describes in detail how to turn on the capability as well as how to author the pages.

Details needed for search

What happens behind the scenes when you create multilingual pages? For one, each language copy is stored in its separate folder. In my case English is the primary site language, and I opted to have Norwegian as a secondary language.

Two pages in the original language.

One of the pages is translated.

Secondly, to connect and keep track of the original vs. the translated pages there are some internal properties in SharePoint per item keeping track of this.

SharePoint Column	Search Managed Property	Description
_SPTranslatedLanguages	SPTranslatedLanguages	Source page stores a list of the language of all translated languages. E.g. nb-no.
_SPTranslationLanguage	SPTranslationLanguage	Translated page stores the language of the page. E.g.: nb-no
_SPTranslationSourceItemId	SPTranslationSourceItemId	Translates page stores a reference to the original page. E.g: 1f50aa8b-d31e-415a-bc4a-4ced38f5cdc4
_SPIsTranslation	SPIsTranslation	The translated page stores the boolean value true to indicate it’s a translation and not the original. The original article has no value.

The desired behavior when searching is to return a page in the users’ display language if it exists. If a translation does not exist, fall back to the original language – which follows the default language of a site set at site creation.

The examples below explain the logic used by Microsoft Search, and you can replicate this yourself as well in custom solutions.

To return English pages returned the query would be:

(SPTranslationLanguage:en-us OR (NOT SPTranslatedLanguages:en-us AND NOT SPIsTranslation:true))

Both pages returned in the original language - English.

To return Norwegian pages the query would be:

(SPTranslationLanguage:nb-no OR (NOT SPTranslatedLanguages:nb-no AND NOT SPIsTranslation:true))

One translated page and one non-translated returned.

If you are using the PnP Modern Search web parts you can use the query variable {PageContext.currentUICultureName} to substitute the correct display language into the query.

Easy as pie :)

PS! For the KQL savvy, you can not replace the NOT operator with the – (minus) operator for boolean values – for some weird reason that fails.

Cover photo by Mat Reding @ Unsplash

Microsoft Search vs. PnP Modern Search – a clash of titans!

Sorry for the clickbait title. I have been meaning to write about how I think about the out-of-the-box Microsoft Search vs. the PnP Modern Search web parts for a while, so here goes.

Disclaimer: I have been working with search for 20 years, enterprise search for 15 years and SharePoint and search for almost just as long. The opinions stated are my own and they are based on my experience with search in FAST ESP, SharePoint and Office 365 over the years and are not influenced by my role as a program manager on Microsoft Search. No money or breaking of fingers were involved in writing this post :)

A little history

With SharePoint 2013, Microsoft shipped many ways to customize how search worked. Both from a technical and user experience perspective. The more important one being the ability to use arbitrary HTML, CSS, and JavaScript in custom display templates with the Search Result web part and the Content by Search web part (CSWP).

Fast forward to SharePoint Online and the release of modern SharePoint pages late 2016. Modern pages came with a new set of web parts and lacked the super-flexible CSWP. Modern pages did introduce the Highlighted Content web part (HCWP), positioned as “...the newer, simplified version of the Content Search web part.”

When customer transitioned from classic to modern a gap had been formed between the CSWP and the HCWP which needed filling. And this is where the awesome sharing in the SharePoint community comes in, put into a sharable system with Microsoft 365 Patterns and Practices (PnP).

The first version of the PnP Modern Search web parts (MSWP) were added as a SharePoint Framework sample by Franck Cornu back in October of 2017. My first contribution to the sample was made in January of 2018 to include additions needed for a customer project I had at the time.

Back then MSWP was one web part; the result web part which included an optional built-in refinement panel. The search box web part got introduced late February of 2018, with the vertical web part following later. My main reason for looking at the sample web part in the first place was the need of a roll-up web part for a project. The Highlighted Content web part (HCWP) lacked the needed templating required – and still does, and why bother develop something if it already exists? Much easier to add to a maintained code base compared to owning it completely on my own #lazyprogrammer.

Image by Zhang Kenny @ Unsplash.

And this is where I believe the PnP Modern Search web parts has its sweet spot – bridge the gap between the classic Content by Search web part and the new Highlighted Content web part. This was my opinion back in 2018, and it still is in 2020. I actually formed the opinion back in 2016. That is 4 years and counting.

Why? you might wonder.

The modern search page was soft launched as part of the SharePoint Home page with the May 4th event in 2016. An out-of-the-box non-configurable search page. The page itself did have a link at the bottom which took you to the classic enterprise search center, but it was indeed the start of mainstreaming the search experience. Microsoft removed the link in March this year without anyone really noticing, which leads to think it was the right thing to do :)

When SharePoint Home launched in 2016, I figured it’s time to stop creating classic display templates. I have a long history creating display templates and you can download some open source ones I started to create back in 2014 from github.com/spcsr. I call this out to emphasize I have indeed customizeded search in the past and thought it a good idea at the time. With the introduction of SharePoint Home the journey for Microsoft Search was started – one experience to work on across desktop and mobile – I changed my mind.

At Ignite 2017 the vision for Microsoft Search as you see it today was launched and the search experience from SharePoint Home made its way over to office.com. Last year, in 2019, Microsoft Search in Bing followed suit. From a slow start, Microsoft Search is hitting more and more client endpoints with the promise of a coherent search experience. Same search box everywhere, with predictable results – all driven by the same Microsoft Search backend. And this will continue to expand going forward.

https://docs.microsoft.com/en-us/microsoftsearch/overview-microsoft-search

Let us go back again four years to 2016. I don’t think myself as very good at predicting the future, but this time I got it right. My take was, if I cannot control which search box people search from with no options to customize the result page they get to, why spend a lot of time and money to create something custom?

Sure, I have the ability and skills needed to tune search both from a relevance and UX perspective to be awesome for a specific business. I ended up thinking it is not worth spending the time or money. I would rather use my consulting hours on other fun and valuable projects. Did I have to fight customers and argue my point? YES!! Many had invested in custom search solution for classic SharePoint and wanted the solution brought over as part of their cloud journey. For those who have been privy to these customers conversations they know the following to be a common Mikael statement:

“If you believe you are right and really really want to create a custom search experience, feel free to do so, but you have to get another consultant to do so. It could be my colleague, but it won’t be me.”

Hindsight I hope they see that I was right and that their money was better spent elsewhere.

If you add the personalized relevance introduced in 2017 based on the Microsoft Graph (then Office Graph) to the equation, it’s just not worth competing with it. Too hard. To me anyways.

Four years after the inception of the Microsoft Search search experience, Microsoft is finally adding capabilities customization capabilities to the search result page. You can add your own verticals, add your own refiners and limit what is returned per vertical (roadmap item 57054). Why did it take so long to get customization functionality...? Well, I do not have all the history as I joined the Microsoft Search team in June 2019. One key part is that Microsoft need to ensure the experience works in multiple clients, not only SharePoint. Therefore some of the classic SharePoint constructs (which were awesome when launched for SharePoint 2013, and still are today in terms of architecture) have to yield as Microsoft Search encompasses multiple clients and content. At the same time as classic search is being left behind I do believe we are also innovating with similar, better and certainly more cloud forward solutions from a technical perspective.

“If you’re so PRO Microsoft Search, why do you then work on, manage and promote PnP Modern Search?”

The answer is simple. Because I love to develop and I do believe PnP Modern Search has a good place in the eco-system for search driven apps, specific portal solutions, or for a more flexible roll-up web part in modern SharePoint pages. The portal/app scenario is available as a downloadable PnP provisioning template.

Did you see PnP Modern Search started to be mentioned on search roadmap slides in 2019? I might have had a finger to do here – and it is because Microsoft do see the value in providing full customization capabilities with search. For scenarios which really really really benefits from special technical and UX capabilities. SharePoint has always been a platform for those needing full flexibility, and it still is via the SharePoint Framework. For the majority of enterprise search needs I don’t believe this level of customization is needed.

Short term the PnP Modern Search web parts can potentially let someone create a better enterprise search center, and I know this happens, but it comes at the cost of educating your users to use a search box you control. An impossible task in my opinion, leading to a disconnected search experience for the end-user as users start their search journey from:

• SharePoint Home
• Office.com
• Teams
• Work, Excel, PowerPoint
• Outlook
• Windows 10
• The browser address bar
• Microsoft Search in Bing
• Web application
• Desktop application
• Mobile applications
• etc.

I honestly believe the out-of-the-box Microsoft Search experiences is the best way forward for most Microsoft 365 users out there, and Microsoft continues to strive to make it better. Not just for the majority of users, but also for the edge cases.

PS! I was about to lose faith in Microsoft Search back in January of 2019, but I’m back on track. Read my post The conspiracy is real! for more details.

If you believe me to be wrong and off my rocker I’m all ears for reasonable arguments to sway me over. Yes, I do acknowledge there are good reasons to control and customize enterprise search, but I find it hard to justify the investments vs. the outcome.

References:

• Microsoft Search
• PnP Modern Search
• Microsoft 365 Patterns and Practices (PnP)

Cover photo by Attentie Attentie @ Unsplash.

One year anniversary as a blue badge at Microsoft

One year ago I joined the Microsoft Search Experiences team in Oslo as a program manager, and time has gone by really FAST (pun intended). Overall it has been an awesome experience - largely due to the great team I’m part of. It has been very different from my previous job as a consultant, thus hard to compare, but I’m not done yet and have things I want to accomplish while I’m still young.

Here’s a list of some of what I’ve been part of or accomplished over the past year – all non-NDA stuff :)

• I’ve written way fewer blog posts compared to the last years, which was expected, but I do hope to get the volume up a bit again.
• Released my Troubleshooting Search in SharePoint Online self-help manual on Amazon
• Two trips to India
• Two trips to the US
• Three trips to Tromsø (northern Norway)
• Written about 4,000 e-mails – yeah, I know that’s not much at all, but it is for me
• Presented at Ignite and SPSOslo (about what I usually do)
• Made a lot of contacts across the v-team which Microsoft Search is across
• Organized a hands & crafts x-mas social workshop for my team
• Worked on multiple features underway – from small to large
• Got the Microsoft Search code base up and running and did multiple pull requests as part of a hackathon week, fixing some real but minor issues
• Won the business value award at an internal hackathon with my fellow hackers
• Worked to align the out of the box Microsoft Search offering and messaging with the PnP Modern Search web parts
• Talked to a good amount of high value customers on various search topics – which is not that different from talking to smaller customers as the search issues and needs are not that different except scale ;)
• Still trying to see if we can expose the Live Persona Card in Office 365 as a re-usable control in SPFx (not given up yet)
• …and more

I’ve interacted and contributed on too many projects to count, and it has proved valuable to bring my SharePoint and Office 365 experience into the team. As Microsoft Search is expanding, SharePoint is still a large part of it content wise and knowing the inner workings when moving from classic to modern is never a bad idea.

Governance tip on finding overshared content in O365/SharePoint

Unless you have put in place the right governance measures you might at some point experience content showing up in search which shouldn't be there. A common problem are items overshared to the Everyone except external users group. Using search itself, you can easily find these items and put in place the right measures to remedy any oversharing.

When someone creates a sharing link on a document in SharePoint Online today, the sharing information is stored in a searchable managed property in the search index named SharedWithUsersOWSUSER. The same information is stored in the internal SharePoint item field named SharedWithUsers.

By issuing the query below as any user (as it’s explicitly shared with everyone and everyone has access) you can find all documents with an everyone sharing link attached to them.

SharedWithUsersOWSUSER:spo-grid-all-users

The same approach can be used with any claim/user/group as documented in the official eDiscovery documentation.

You can also try the below query to find items which are shared:

ViewableByExternalUsers:1 OR ViewableByAnonymousUsers:1

If you get the refiner values from SPSiteURL, you can aggregate all sites which have shared content.

If you want to remedy oversharing to everyone, you can turn off the showing the Everyone claim with SharePoint Online PowerShell:

Set-SPOTenant -ShowEveryoneExceptExternalUsersClaim $false

or with PnP PowerShell:

Set-PnPTenant -ShowEveryoneExceptExternalUsersClaim $false

This merely hides the claim from showing, so you might still want to run scheduled reports to find items which are shared with everyone in case you want to prevent this behavior.

Happy sharing!

Image by Markus Winkler @ Unsplash

Searching within Office 365 Groups or Teams content

There are at least three ways to limit the search results within a group or a team using the keyword query language (KQL). These are cases where the managed property used exists on all items for the site/group.

Path

If you know the URL of the groups site you can use a path filter, and remember the trailing slash (/) to avoid edge case inclusion of other data.

path:https://contoso.sharepoint.com/teams/myteam/

Site Id

If you know the underlying site id of the Groups site.

SiteID:<guid>

Group Id

If you know the Group Id, meaning the AAD id of the group, you can use this as well.

GroupId:<guid>

Bonus: Search within all the groups a user is member of

And here’s the kicker to effectively scope results to all the groups a user is member of. Say you have pulled out a list of all the groups a user is member of via the Microsoft Graph API:

https://graph.microsoft.com/v1.0/me/memberOf/$/microsoft.graph.group
?$filter=groupTypes/any(a:a eq 'unified')

Armed with all the group ids, you can issue a SharePoint Search REST query using a wildcard (*) as the query text and the following FQL in the refinementfilters property.

GroupId:or(<guid1>,<guid2>,….)

GroupId:or(81eb706e-2904-4ac6-89f4-a0cf9d59d1c4,79958190-024b-4c62-ab55-65dc9a066cac)

If you are a member of many groups, then I suggest using a POST payload instead of GET, and refinementfilters is not limited by 4k either (at the time of this writing) :)

See https://www.techmikael.com/2013/07/limiting-search-results-by-exact-time.html for an explanation of refinement filter usage.

Note: How the above syntax will work using the Microsoft Graph Search API which is in beta, is not known at this point.

Photo by Mike Szczepanski @Unsplash

I finally found a use case for the Surface Pen (after 4 years)

I’ve been using Surface Pro’s since the Pro 3 and I know have a Pro 6. I have always liked the form factor of it and each one of them came with a Surface pen. Sure, I’ve tested the pen for fun and used it occasionally to sign something, but in the end I’m a laptop guy, not a tablet guy when it comes to work. So the pens have trustfully followed along in my back back never being used.

Until now that is!

In my home office I have opted to get a speaker phone as wearing a headset for longer periods of time is uncomfortable. The advantage with cabled headsets is that they have a mute/unmute button on the cord, allowing you to sit back and toggle the microphone on and off based on the conversation.

With a speaker phone I can still mute and unmute. Doing so requires me to stretch over to the speaker, not having the button easily in my hand. When using Microsoft Teams, I can use a keyboard shortcut shift-alt-m, but that requires the application to be active and is a bit too quirky.

You probably see where this is going :)

The solution is pretty simple. Being a programmer I browsed around and found out how to programmatically toggle a microphone on/off. You can find the C# code over at https://github.com/wobba/MuteToggleMicrophone. (I did look at NIRCMD, but it mutes speaker as well as mic when devices have the same names.)

As the program is a console application, I browsed to the .exe file, and created a shortcut for it as shortcuts can be set to be run minimized. The .exe takes the parameter of a string matching the name of your microphone device (if you have more) or “all” to toggle each one.

The last step is to configure pressing the eraser button on the Surface pen to launch this program.

Now I can relax back in my chair, not touching the keyboard, and mute/unmute myself as needed :)

Modern Script Editor web part now with support for Teams tabs!

Yeah, you got it, you can now use the modern script editor web part as a poor mans solution to bootstrap applications as a Teams tab. And the better approach is to develop a real Teams tab application – just so you know!

I did not really intent to do this, but I had a request to update the web part to SPFx v1.10.0 as some people have issues building SPFx v1.4.1 on their machines. While upgrading I figured I’d do a revamp so I updated the web part with a proper html editor and enabled Teams tab support as well – because why wouldn’t you right? (A shoutout to the PnP team for suggesting this).

Check out the readme from the below link if you want to test the Teams tab support as you need to edit some default settings to make that happen.

https://github.com/SharePoint/sp-dev-fx-webparts/tree/master/samples/react-script-editor#support-for-teams-tabs

Side effect of mapping crawled properties to refinable managed properties

Image by Miguel Perales @ Unsplash

Mapping a crawled property to a refinable one is a common scenario in SharePoint search. If you expect the content behind the crawled property to also be full text searchable, you need to map the crawled property to an additional managed property which is set to be Searchable.

Explanation

By default a crawled property is full text searchable. Meaning if the property contains the text “pnp is awesome”, then a query for the term pnp, will yield a result.

Once you map the ows_columnName crawled property to say RefinableString00, then the above query will not work as RefinableString00 is not searchable by default, and overrides the behavior of the crawled property. If you query for RefinableString00:pnp as a property query you get results as RefinableString00 is queryable.

If you want the content to be both full text searchable and to be used in a refiner, create a new managed property, for example named TotalRecall, and map ows_columnName to this property as well. Remember that a crawled property can be mapped to any number of managed properties, and something you might want to do especially when mapping them to refinable managed properties.

For crawled properties applying to user profiles you can map to HiddenContents, which acts in a similar way. See https://www.techmikael.com/2014/11/how-to-make-sure-user-profile.html for additional details. 

References:

• https://www.techmikael.com/2017/01/clarification-on-automatic-managed.html
• https://www.techmikael.com/2014/07/what-makes-sharepoint-column-searchable.html
• https://www.techmikael.com/2014/12/solution-to-cannot-search-content-in.html

Set a document to open in view-only mode by default

Document productions and lifecycles come in many shapes and fashion. One scenario is sharing a document without the intent for the reader to necessarily make any edits or comments. To reduce the footprint of auto versioning and accidental edits of a document you have the ability to explicitly make the document open in view-only instead of in edit mode.

Other people may still edit and comment the document, but they must first make an explicit choice to do so.

If you didn’t know about this easy tip, here’s a video showing how you enable it using Word Online. The same setting is also available via desktop clients. If you already knew how, then you’re all set :)

Retrieving thumbnails/previews for SharePoint files and pages via Microsoft Graph API’s

Back in 2011 I wrote a post about thumbnail previews in search via the Fast Search for SharePoint which relied on Office Web Apps to provide the actual thumbnails. Fast forward to 2020, visualizing a page or document in a roll-up scenario is still valid, and fortunately todays service allows much better resolution on the thumbnails.

The API itself is coming via the Microsoft Graph and works on driveItems and is well documented at https://docs.microsoft.com/en-us/graph/api/driveitem-list-thumbnails.

If you want to test this out yourself, download the SPFx sample from https://github.com/SharePoint/sp-dev-fx-webparts/blob/main/samples/js-msgraph-thumbnail/README.md to get started.

Digging into the API’s, the approach I have chosen to go from a SharePoint item to a Graph item is to use the site id, list id and item id – all id’s easily available per SharePoint item, either via REST API’s or the search API, using the calling signature below:

<endpoint>/sites/<site id>/lists/<list id>/items/<item id>

/driveItem/thumbnails/0/<custom size>/content

The above URL can be used directly in an <img> tag, removing the need for multiple calls in order to get the actual image.

Token	Description
endpoint	graph.microsoft.com/v1.0 (https://docs.microsoft.com/en-us/graph/api/overview?view=graph-rest-1.0) tenant.sharepoint.com/_api/v2.0 (https://docs.microsoft.com/en-us/sharepoint/dev/apis/sharepoint-rest-graph)
site id	Site object id – GUID.
list id	List object id – GUID.
item id	ListItem object id or unique id – Integer/GUID.
custom size	See Requesting custom thumbnail sizes in the official Microsoft Graph documentation.

If you choose to get the drive item in a different way, that’s all good, as the thumbnail API itself would be the same. In addition to using custom sizes for the preview, there are also some default sizes you can use, all documented.

The above approach have been verified for most common file types (except Excel) as well as for modern pages.