Scenario:

Managed service accounts passwords expired.  Access to part of Central Administration are no longer accessible.  Sites are starting to go down because app pool passwords are managed accounts and have expired.

Soapbox moment:

Firstly, there is NO real excuse for this in SharePoint 2010 because the ability to have this done automagically for you is BUILT-IN, so either your farm admin is so over taxed (usually the case) or incompetent (the two aren’t mutually exclusive). 
I take the liberty to say all of this as someone who has had this happen to them, otherwise I wouldn’t be able to write about it, right?

Resolution:

To start, you aren’t going to be able to do anything with SharePoint until you can get the Timer Job Service running again because everything is driven by timer jobs. 
Using a credential that has full admin rights to the box and is a Farm admin, change the account the Timer Job Service runs as and start the service.  This must be done on all servers in the farm. 
Don’t fret, the next things you are going to do is fix this back to the way it should be but you can’t do the next steps without the timer job service running, so just play along.

Go to http://www.yourserver.com/_admin/ManagedAccounts.aspx and edit your farm service account and tell it to change the password now.

Go to http://www.yourserver.com/_admin/FarmCredentialManagement.aspx and select Farm Account.  You will see your registered service account (the one that you just changed the password for) and click ok.  This will go reset your Timer Job Service account to the registered account which now is active and working.
Next, create a text file called on your server which will be a list, one account per line, of service accounts that you are going to have auto-updated.

Then using the SharePoint 2010 Management Shell interface change the passwords and set to auto change run using this variable script:

foreach ($account in Get-Content driveletter:filename.txt)
{
Set-SPManagedAccount -Identity $account -AutoGeneratePassword -PreExpireDays n#days -Schedule “monthly between n#dayofthemonthvalue hh:mm:ss and n#dayofthemonthvalue hh:mm:ss” -confirm:$false
}

The script that I actually used, without the variables looks like this:

foreach ($account in Get-Content c:managedaccounts.txt)
{
Set-SPManagedAccount -Identity $account -AutoGeneratePassword -PreExpireDays 30 -Schedule “monthly between 7 02:00:00 and 7 03:00:00” -confirm:$false
}

Once this command has completed successfully you will see that your last password change just happened and that your next password change is scheduled.  Make sure that if your next scheduled password change isn’t in conflict with a password change minimum group policy that won’t allow passwords to be changed before a minimum number of days or you will end up with some errors in your ULS Logs and some misfired password change attempts.

Lastly, go to http://www.yourserver.com/_admin/FarmCredentialManagement.aspx and walk through all of the farm credentials and let the accounts get synced up.  This should re-spin up the app pools and get your users back into the site, but if not, do an IISRESET and things should be back online.

Shout outs

Huge thanks to my partners in crime on this one, Derek Martin and Trent Foley of Slalom Consulting, for helping with the out of the gate perfect PowerShell scripts that are referenced above.

While I was busy figuring out how to break back in to Central Admin, they figured out the proper script to reset the passwords and set the auto change programmatically.  This script can be used in advance of this type of shenaniganal activity to ensure that while you are building your farm you get this set right the first time and not have to do it manually (which is often the excuse when you are using 30+ managed accounts in a farm).