It’s a small, dedicated XenApp 6.5 farm, consisting of just 2 servers serving Published Desktops to around 50 concurrent users. They did qualify as Power Users for sure, with some of them having iexplore.exe’s running at 1 GB RAM.

They connect over some kind of MPLS network to a data center, with a dedicated print server next to the XenApp servers connecting users to a bucket load of printers over the MPLS link.

I’ve “rebuilt” the servers twice already. The first time was the result of some storage-related issues. I was able to revert back to a snapshot, or to do it correctly: setup clones based on a previous snapshot. Users were able to log on again.

After a number of weeks, the users reported that they could not log on again, stating “low on system resources”. We rolled back to a weekly snapshot, buying us some time to install 2 new servers from scratch, to rule out some cloning-related issues.

We moved all users to the new servers and all was good yet again. But that wouldn’t stay that way obviously, otherwise, this would be a rather lame article, wouldn’t it?

All of a sudden, users started logging on with Temporary Profiles. After ruling out the usual suspects, the issue started to become very complicated. A temporary profile was loaded even with a single user on that server, even after reboot. We got reports from different users, different timings… so we had a “random” error on our hands.

Since the servers were rebuilt from scratch we were now sure that reverting to snapshots would only temporarily fix the problem. At that point, we were sure it would return we just didn’t know when.

We decided to log an MS Support case and after a few case engineers gave it a shot, I was pointed in the right direction: registry size. To be more specific: the size of the default Local System registry profile. It turned out that the size of the DEFAULT file had grown to a staggering 1.8 – 2GB in size.

We all know that a Default User Profile is only used when a user doesn’t have a profile and a new one needs to be created. That’s still true of course, but .DEFAULT does not stand for the Default User Profile as I initially assumed (and wasn’t corrected by MS Support). Thanks to Aaron Parker (@stealthpuppy) who pointed this out to me, this is actually the Local System Profile. Still, a 2Gb Local System profile would consume way too much system resources obviously.

So now we did find our bad guy, and MS provided a procedure to restore a DEFAULT file:

1. Boot one working machine into recovery console and copy the file c:\windows\system32\config\default to the root of the hard disk. Reboot normally
2. Copy the default file from the working machine to the failing machine to the root of the hard disk
3. Boot the failing machine using recovery console and copy the file c:\windows\system32\config\default to c:\windows\system32\config\default.old to have a backup.
4. Copy the file c:\default to c:\windows\system32\config\default
5. Reboot the machine

My tip: make a copy of the DEFAULT profile anyway after initial setup. That way you can always revert back to that one quite easily.

Next up: the search for the root cause. Using a tool provided by MS Support we were able to identify 2 registry keys that were clearly oversized.

HKEY_USERS.DEFAULT\Printers\DevModePerUser

HKEY_USERS.DEFAULT\Printers\DevModes2

First of all, you can just delete those keys or their content. Users will not have any impact, but the file size will remain the same, only with a lot of “white space”.

MS added some details to this: they did have some “rare cases” where print drivers attempt to write into the default profile, filling it up until the point where the registry is becoming too large causing registry pressure.

Guess what… we had such a “rare case” on our hands. Initially, I restored a lean and mean copy of the DEFAULT file, only to see it grow to 1.15GB in just 2 weeks. After another restore, I set a new XenApp policy: store printer preferences in user profile only (instead of “client device first, user profile second”). For a few weeks after that, file size stayed stable.

But this wasn’t the end of it, file size did grow, although slower than before. Starting with a clean DEFAULT file I was able to find the reason behind everything: a number of printer drivers. When a user connects to a printer, that connection adds 2 registry entries, see screenshots below.

On some printers, this happens only once. But for some others, this happens every time! In this specific case, every time a user logs on, a GPO connects the printers… and 2 registry entries are added. Again, this is driver-specific, meaning that some drivers do show this behavior while others don’t.

In this specific customer case, a majority of printers are using the same driver. And look what I found in the release notes of the latest update:

Fixed:

– When the driver is reinstalled it creates an additional instance of an existing registry key instead of just re-using it. Each time the driver is re-installed a new key is added to the registry.

Looks promising? Well, it has proven to be more than promising. After this update, this specific driver didn’t exhibit the weird behavior anymore. For those who want to know, it’s a RICOH SP4100N PCL5 driver.

We do still have 3 printers that use a driver that’s updating the default profile for some reason. There is no driver update available and I stay clear of so-called universal drivers. But they are on their way out anyway in a few weeks or months time, so at this point, the environment is stable, the file size doesn’t grow dramatically anymore and we just keep monitoring the file size and restore a clean version when it might become necessary.

Update: Hotfix available

Microsoft has released a hotfix for this issue: http://support.microsoft.com/kb/2871131.