Issue: Starting a service crashes SharePoint site with unknown error

Issue:

After you stop a service (such as User Profile Synchronization or Search service) and try to start it again in Central Administration, the SharePoint 2010 site pages will crash with an unknown error.



Analysis:

Prior to the issue, the password was changed for the services account in SharePoint. However all services were working.

Searching the cascading numerous errors in ULS to find the root cause kept my head spinning for hours. I knew that issue was because of the password change in services account but couldn't spot where is the start point to fix.
Some of the error messages I found in IIS log:
- "Application pool 15f65d1adf954fa68aee81f7bcb9cec8 has been disabled. Windows Process Activation Service (WAS) encountered a failure when it started a worker process to serve the application pool."
- Another message is that the account may need batch logon permissions.

 then I had a clear image of what happened:

1. The service (User Profile Synchronization service or Search service) tries to start
2. Service fails to start because its application pool in IIS could not login 
3. Since its application pool is down, it will bring down all the services that are running on it. If User Profile service is working on this application pool then it will stop. Also if Metadata service is working on this application pool then it will stop, and bring down the Profile service as well since User Profile depends on Metadata.
4. All content on SharePoint pages (such as pages, web parts,..) requires a valid user login to display according to user permissions. Since User Profile service is down, current user cannot be verified and SharePoint shows unknown error.

I already made sure that that the service account has the necessary permissions and that services and service applications are setup correctly on CA. Also the application pool account is using the correct password.
Well, it didn't work! Every time I try to start the application pool in IIS it stops after couple of seconds. Agrrrrrr...

It appeared that the faulty application pool "15f65d1adf954fa68aee81f7bcb9cec8" in IIS becomes corrupted object and will not work.



Solution:

Relocate applications that are on corrupted application pool into another one with exact same settings.
First, make a backup for WFE and APP servers.
Depending on your current configurations, make the following changes in IIS in WFE and APP servers:

1.      Create new AppPool “MetadataCustomAppPool” as copy of the corrupt AppPool but using new password  for service account
2.      Move “metadata” and “Profile” applications from corrupt AppPool to   “MetadataCustomAppPool” AppPool.  This is useful when other service is down (and     subsequently its AppPool is down too), it will not affect the Metadata/User Profile and the   SharePoint site will continue to work.
3.     If other service applications are in corrupt AppPool, then you can relocate them into MetaDataCustomAppPool or create a new AppPool for them as appropriate.
4.      Restart IIS

Performance problem: SharePoint site is very slow

Performance Problem:

SharePoint 2010 Server site and Central Admin are unrealistically too slow (16 seconds or more) to open in browser, without showing any error. You added much more server hardware resources (CPU, RAM and HDD) but this solution fails to improve performance.

Analysis:

In IE, hit F12 key and found that the site home page took 73 seconds to load!

In Central Admin, click on the Health Analyzer to find an interesting item in the performance section: “Application pools recycle when memory limits are exceeded.”

According to this Microsoft article here:

Application pools recycle because memory limits have been enabled and exceeded. Recycling based on memory limits is not usually necessary in a 64-bit environment, and therefore recycling should not be enabled. Unnecessary recycling can result in dropped requests from the recycled worker process and slow performance for end users who are making requests to the new worker process.

In faulty server -> IIS Manager -> Application Pools -> select the ‘SharePoint – 80' -> right-click -> click Recycling. We see the following 2 values:

·         Virtual memory usage (in KB): is checked to use 1024 KB

·         Private memory usage (in KB): is checked to use 1024 KB

Cause:

Apparently the application pool (SharePoint – 80) is recycling each time it reaches the memory limit as set above to 1024 KB. This happens many times during page loading so the total time needed to load the page is too long.

Solution:

Follow this procedure:

1.   Current user account must be a member of the Farm Administrators group.

2.   Identify the failing server(s) and repeat the following steps on each failing server

[Central Administration -> Monitoring -> click Review problems and solutions, and then find the name of the server in the Failing Servers column, example: SharePoint - 80].

3.   Current user account must be a member of the Local Administrators group on each server in step 2 above.

4.   In faulty server -> IIS Manager -> Application Pools -> select the ‘SharePoint – 80' -> right-click -> click Recycling. Clear the 2 check boxes of Virtual memory usage (in KB) and Private memory usage (in KB) -> click Next -> click Finish. (Repeat this step for all AppPools of slow WebApps).

Open the site again, and now it’s really fast!