Wednesday, December 11, 2013

Duplicate Trimming in SharePoint 2013 is causing confusion

[Update - Verified with July 2014 CU]
You can now turn off security trimming via the Query Builder on your search web parts.

  • Edit web part
  • Click "Change query"
  • Click the "Settings" tab
  • Toggle "Don't remove duplicates'

[Original Post]
Duplicate trimming as a function in search is a good idea. The intent is to reduce noise by discarding duplicate or equal items in a search result. The issue with SharePoint 2013 is that trimming is implemented too coarse and a lot of good results are hidden for the user. Also, turning off duplicate trimming is not an edit web part task as the option is hidden in a JSON property on the web part.

My recommendation at the moment is to turn off duplicate trimming, and if users complain about real duplicates being show, tell them to clean up the data. Most of the time you really don’t want duplicates of items/documents stored anyways.

I’ll dig into and explain more about how duplicate trimming is performed in SharePoint 2013 in a later post.

If you are on-premises you may use the same procedure as I used in Make sure your People Search is fuzzified, where using PowerShell, you modify the internal JSON property. Using the same script change line 13 to read:

$dataProvider.TrimDuplicates = false

Being inspired by Chris O’Brian’s post on using CSOM with PowerShell I have modified my code to use PowerShell  and CSOM. By changing the credentials line, you may use the code against both SharePoint on-premises and SharePoint on-line.

# Author: Mikael Svenson - @mikaelsvenson
# Company: Puzzlepart
# Date: December, 2013
# Reference:

# replace these details (also consider using Get-Credential to enter password securely as script runs).. 
$username = "" 
$password = "password" 
$url = ""
# the path to the SharePoint Client dlls' 
$dllPath = "D:\SP2013-dll\ISAPI\"
$securePassword = ConvertTo-SecureString $Password -AsPlainText -Force 
Add-Type -Path "$($dllPath)Microsoft.SharePoint.Client.dll" 
Add-Type -Path "$($dllPath)Microsoft.SharePoint.Client.Runtime.dll" 
Add-Type -Path "$($dllPath)Microsoft.SharePoint.Client.Publishing.dll"
Add-Type -Path "$($dllPath)Microsoft.SharePoint.Client.Taxonomy.dll" 
# connect/authenticate to SharePoint Online and get ClientContext object.. 
$clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($url) 

#$credentials = New-Object System.Net.NetworkCredential($username, $securePassword) 
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $securePassword) 
$clientContext.Credentials = $credentials 
if (!$clientContext.ServerObjectIsNull.Value) 
    Write-Host "Connected to SharePoint site: '$Url'" -ForegroundColor Green 

$web = $clientContext.Web
# get guid of the default Pages library to cater for localization
$pagesGuid = $web.AllProperties.FieldValues["__PagesListId"]
$list = $web.Lists.GetById($pagesGuid)
# get localized server relative url
$url = $list.RootFolder.ServerRelativeUrl

$page = $web.GetFileByServerRelativeUrl($url +"/results.aspx");

Write-Host "Checking out page" -ForegroundColor Green 
catch{ Write-Host "Page already checked out" -ForegroundColor Yellow}
$wpm = $page.GetLimitedWebPartManager([Microsoft.SharePoint.Client.WebParts.PersonalizationScope]::Shared) 
for ($i=0; $i -lt $wpm.WebParts.Count; $i++)
    $item = $wpm.WebParts.Item($i)
    if( $item.WebPart.Title -eq "Search Results" ) {
        Write-Host "Found result web part" -ForegroundColor Green 

Write-Host "Turning off trimming of duplicates" -ForegroundColor Green
# Read JSON properties and convert to an object
$dataProvider = ConvertFrom-Json $item.WebPart.Properties["DataProviderJSON"]
$dataProvider.TrimDuplicates = $false
# Convert the object back to a JSON string
$item.WebPart.Properties["DataProviderJSON"] = ConvertTo-Json $dataProvider -Compress
Write-Host "Checking in and publishing page" -ForegroundColor Green 
$page.CheckIn("Modified Search Core Results web part", [Microsoft.SharePoint.Client.CheckinType]::MajorCheckIn)
$page.Publish("Modified Search Core Results web part")