Scenario:

Small Farm 3 tiered topology using Windows with Claims implementation aggregating AD with a custom LDAP database to create the claims roles.

Symptom:

Users of a SharePoint 2010 site get access denied to a site they could access earlier in the day.  As the day goes on, the number of users effected increases.  Eventually only users with full control policies can access the farm.

ULS Log error:

An exception occurred in Custom Roles claim provider when calling SPClaimProvider.FillResolve(): The underlying provider failed on Open..

Root Cause:

The 10 hour default session timeout for the user’s claim has been exceeded and the database housing the Role Data is no longer accessible.   In this case it was due to an expired SQL account password. Changing the password and updating the connection string or just unchecking the password expiry flag in the SQL account will resolve the issue.

Notes from the field:

There was one easy way to prevent this type of user facing outage.  Don’t allow SQL accounts to expire.  EVER.  They are horrible to diagnose because access to the SQL Server is still operational and access using AD authentication is going to throw you off the scent because the main farm access is still available.

Read Scot Hillier’s blog on “Authorization Failures with Claims-Based Authentication in SharePoint 2010”.  Really useful stuff in there about how claims works and extending the timeouts.