Search cannot see some metadata tagged documents SharePoint

Greetings to fellow SharePoint developers and admins.. here is one more issue about crawling managed metadata terms and how to solve it.


Background:
The objective was to have a custom document search by terms. In SharePoint 2013, we have a document library with couple of managed metadata fields (some are single while others are multi value). Based on some logic, a custom code solution will assign (tag) term values to documents. Tagged documents are ultimately to be found using a custom search page by passing 2 term names and their Guids. Example:

/Pages/customsearch.aspx?Sites=5915df2d-f1e7-4247-90ff-9b55a391afc1&SiteCategory=3edf1303-7fd3-497d-a23e-bceb01015f8c

The search page contains Content Search Web Parts. Each one contains a custom query that uses query strings to retrieve all items tagged with multiple values of either or both of the 2 terms used. Example:

{|owstaxIdSites:{QueryString.Sites}*} {|owstaxIdSiteCategory:{QueryString.SiteCategory}*}

The Issue:
The CSWP query show some documents, but do NOT show other documents even if they were tagged with exactly the same terms.

Here is the list of symptoms:
- Document item is tagged with valid terms (we can see/edit the terms with no issue in SharePoint UI in DispForm/EditForm views respectively on all fields)
- Document is crawled successfully and can be viewed by default search page when search by title
- No issue from crawl log regarding said documents themselves nor their metadata terms
- If I just just edit the document and Save, then run crawl it might fix the issue for some documents!
- Also, if I call SystemUpdate() on item from code (without making any other changes), then crawl it will be fixed as well!
- I noticed that for affected documents, all terms from all metadata fields are not showing under under any query.


Failed attempts: 
Tried the following methods,but Non of the below worked:
- Tried re-run Incremental Crawl.
- Tried partial re-index target site or content source (to force search to consider documents as 'dirty' as if the property 'vti-searchversion' was updated) then run Incremental-Crawl.
- Tried Full crawl.

Hmm, so it's not about the search crawl, not the document, not the terms, but something with how the terms are stored in metadata columns. But from the UI, everything seems normal..


Root Cause:

So I paid another visit to the custom code part. After careful checking, I found a bug in code where there are cases where it inserted multiple copies of the same term in one of the fields.

Example: from browser the metadata column show one term 'POLICIES', but from code it has multiple duplicates:

POLICIES|93aef6f0-5558-4c55-9fb2-fbe622a59e8c;

POLICIES|93aef6f0-5558-4c55-9fb2-fbe622a59e8c;

POLICIES|93aef6f0-5558-4c55-9fb2-fbe622a59e8c

Apparently, when multiple copies of the same term are pushed by code to a managed metadata column, then it's it is a column validation issue. The SharePoint UI will not complain but the search crawler somehow will have a problem with it and will drop the managed metadata crawling for the all managed columns of that item altogether. The document item itself is with other non-managed columns will be crawled.

On the other hand, I can conclude the default SharePoint metadata controls offer some validation/fix. When editing the document from UI, and just click Save without making any changes, the duplicate terms might get removed.

Solution:
Check you code to see where it tries to update managed metadata fields with duplicate copies of the same term. No exception will be thrown so you have to carefully check your logic.

FYI, This is a related post where merging terms in store lead to the same issue

User Profile JobTitle Property Returns Null

Problem:

Using SharePoint Object Model, the JobTitle property will always return NULL value for all users, even when users have valid user profiles with available Job Title property and is accessible from CA.

Example: I highlighted the code where JobTitle returns Null:

bool UserExist = uprofileManager.UserExists(user.LoginName);

if (UserExist)

{

UserProfile userProfile = uprofileManager.GetUserProfile(user.LoginName);

returnValue = (String)userProfile[PropertyConstants.JobTitle].Value;

}

While the same user has a Job Title value in his user profile:

Digging for Answers:

- First, I made sure that Job Title property accessibility is set to 'Everyone' in Central Admin
- I elevated the code privileges by using SPSecurity.RunWithElevatedPrivilages code block
- Also while debugging the code, I made sure the user.LoginName variable is in format "Domain\Username"
- Also, found that code to read SPS-JobTitle property would return Null too:
returnValue = (String)userProfile["SPS-JobTitle"].Value; // returns null

- However, all other user profile PropertyConstant properties (like Office, HomePhone,..) are readable correctly for the same users:

PropertyCollection props = uprofileManager.Properties;

UserProfile userProfile = uprofileManager.GetUserProfile(user.LoginName);

foreach (Property prop in props)

{

    Console.WriteLine(prop.Name.ToString() + "=" + userProfile[prop.Name].Value);

}

I concluded that its not the code that has issues, so there must be something with 'Title' and 'SPS-JobTitle' properties preventing the code from reading them.

We know that Title and SPS-JobTitle are special properties linked to Metadata keywords when profile import takes place from Active Directory. The Metadata service is running on 2 servers: WFE & APP. From Central Administration, I can access the Metadata service normally, and see all the Enterprise keywords (System -> Keywords) in Metadata Term Store without an issue.

Okay, now let's have a look on ULS logs. I found errors & highlighted the important lines:

1) Error in Topology:
SharePoint Web Services Round Robin Service Load Balancer Event: EndpointFailure Process Name: UserProfileImport Process ID: 10560 AppDomain Name: UserProfileImport.exe AppDomain ID: 1 Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:02cb72f4a15149bd848db2ba8ab1f055#authority=urn:uuid:235674f601ec4ef1ae9cb964c49bf0d6&authority=https://usascsppro03-nt:32844/Topology/topology.svc Active Endpoints: 1 Failed Endpoints:1 Affected Endpoint: http://usascsppro04-nt:32843/02cb72f4a15149bd848db2ba8ab1f055/MetadataWebService.svc


2) Then error in Manages Metadata service:

Failed to get term store for proxy 'Managed Metadata Service'. Exception: System.ServiceModel.ServerTooBusyException: The HTTP service located

 at http://usascsppro04-nt:32843/02cb72f4a15149bd848db2ba8ab1f055/MetadataWebService.svc is too busy.  ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.   

 at System.Net.HttpWebRequest.GetResponse()    

 at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     -

 -- End of inner exception stack trace ---    Server stack trace:    

 at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)   

 at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)   

 at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)   

 at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)   

 at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)   

 at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)   

 at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)   

 at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown

 at [0]:    

 at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   

 at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)   

 at Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication.GetServiceSettings(Guid rawPartitionId)   

 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass2f.<ReadApplicationSettings>b__2e(IMetadataWebServiceApplication serviceApplication)   

 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass2c.<RunOnChannel>b__2b()   

 at Microsoft.Office.Server.Security.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)   

 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.<>c__DisplayClass2c.<RunOnChannel>b__2a()   

 at Microsoft.Office.Server.Utilities.MonitoredScopeWrapper.RunWithMonitoredScope(Action code)   

 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.RunOnChannel(CodeToRun codeToRun, Double operationTimeoutFactor)   

 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.ReadApplicationSettings(Guid rawPartitionId)   

 at Microsoft.SharePoint.Taxonomy.MetadataWebServiceApplicationProxy.GetTermStoreId(Guid rawPartitionId)   

 at Microsoft.SharePoint.Taxonomy.Internal.DataAccessManager.GetTermStoreData(MetadataWebServiceApplicationProxy sharedServiceProxy)

3) Then a warning:

The managed Metadata Service ‘Managed Metadata Service’ is inaccessible.

Interestingly, when browsing the Metadata service in IIS:
- MetadataWebService.svc is not available in WFE server (HTTP Error 503)
- MetadataWebService.svs is available in APP server


Root Cause:

It appeared that this all related to an issue Managed Metadata service leading to the issue in SharePoint topology (load balancer) service officially known as "Application Discovery and Load Balancer Service Application".

The Managed Metadata service was setup to work on both WFE and APP servers. on WFE, the Metadata was not working as we saw the HTTP error 503. But that was not a problem since the other Metadata service works fine on the APP server, hence the perfectly fine keywords & user profile accessibility. However, the topology (load balancer) service was having an issue, because it basically round-robin requests to services if we have more than one server to offer these services. Since the Metdata data service is not working in WFE then it's end-point will not be valid. But topology service still thinks it is a perfectly valid end-point and keep trying to connect to the Metdata service through that faulty endpoint, until a connection time-out occurs and a 'server unavailable' exception is thrown in log.

Solution:

1. Stop the faulty Managed Metadata service (in this case on WFE) from CA             
2. In some cases, you need to stop/start Metadata service on other server too to refresh the endpoints
3. Stop then restart the application pool of faulty Topology application from IIS (in this case on WFE)
4. Make sure all application pools are started normally

Result:

After applying steps above, the Metadata term store & User profiles work normally. Code able to read the values for JobTitle property for all users.

Useful resources:
http://social.msdn.microsoft.com/Forums/sharepoint/en-US/869fb269-2a6b-4efa-a0e0-c4a829065ca4/load-balancer-error-mesaage?forum=sharepointadminprevious

http://blog.henryong.com/2011/10/18/sharepoint-2010-intermittent-search-errors-web-services-round-robin-service-load-balancer-event-endpointfailure/

http://www.mysharepointadventures.com/2012/03/event-8313-topology-load-balancer-endpointfailure-searchservice-svc/

http://blogs.msdn.com/b/russmax/archive/2010/05/06/sharepoint-2010-shared-service-architecture-part-2.aspx

https://adammcewen.wordpress.com/2012/12/14/redirecting-user-profile-sp-job-titles-to-mmd-term-set/

Issue: Starting a service crashes SharePoint site with unknown error

Issue:

After you stop a service (such as User Profile Synchronization or Search service) and try to start it again in Central Administration, the SharePoint 2010 site pages will crash with an unknown error.



Analysis:

Prior to the issue, the password was changed for the services account in SharePoint. However all services were working.

Searching the cascading numerous errors in ULS to find the root cause kept my head spinning for hours. I knew that issue was because of the password change in services account but couldn't spot where is the start point to fix.
Some of the error messages I found in IIS log:
- "Application pool 15f65d1adf954fa68aee81f7bcb9cec8 has been disabled. Windows Process Activation Service (WAS) encountered a failure when it started a worker process to serve the application pool."
- Another message is that the account may need batch logon permissions.

 then I had a clear image of what happened:

1. The service (User Profile Synchronization service or Search service) tries to start
2. Service fails to start because its application pool in IIS could not login 
3. Since its application pool is down, it will bring down all the services that are running on it. If User Profile service is working on this application pool then it will stop. Also if Metadata service is working on this application pool then it will stop, and bring down the Profile service as well since User Profile depends on Metadata.
4. All content on SharePoint pages (such as pages, web parts,..) requires a valid user login to display according to user permissions. Since User Profile service is down, current user cannot be verified and SharePoint shows unknown error.

I already made sure that that the service account has the necessary permissions and that services and service applications are setup correctly on CA. Also the application pool account is using the correct password.
Well, it didn't work! Every time I try to start the application pool in IIS it stops after couple of seconds. Agrrrrrr...

It appeared that the faulty application pool "15f65d1adf954fa68aee81f7bcb9cec8" in IIS becomes corrupted object and will not work.



Solution:

Relocate applications that are on corrupted application pool into another one with exact same settings.
First, make a backup for WFE and APP servers.
Depending on your current configurations, make the following changes in IIS in WFE and APP servers:

1.      Create new AppPool “MetadataCustomAppPool” as copy of the corrupt AppPool but using new password  for service account
2.      Move “metadata” and “Profile” applications from corrupt AppPool to   “MetadataCustomAppPool” AppPool.  This is useful when other service is down (and     subsequently its AppPool is down too), it will not affect the Metadata/User Profile and the   SharePoint site will continue to work.
3.     If other service applications are in corrupt AppPool, then you can relocate them into MetaDataCustomAppPool or create a new AppPool for them as appropriate.
4.      Restart IIS