Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load
On a recent project, the customer experienced an issue where updating the Citrix Provisioning Services (PVS) image to Internet Explorer (IE) 11 caused a major outage.
If any users caused two Microsoft Office applications to run simultaneously, the CPU would spike between 80% to 100%, and the server would become unresponsive. The only recourse was to use the hypervisor to either Force Shutdown or Force Reset the server.
The customer’s previous PVS image used the same version of Microsoft Office and other core applications. The only thing that changed was an upgrade from IE10 to IE11. After reverting users to different servers for the Office applications, troubleshooting the root cause began.
To make a very long story short, it was NOT:
- An IE11 issue
- A PVS issue
- A XenApp issue
- An Office application issue
- Or a user issue
It turns out there is a known issue on Server 2008 and Server 2008 R2 with the NTFS IO system being unable to handle the stress of large amounts of file IO. Both Citrix and Microsoft were involved in the troubleshooting and reading of dumps. Microsoft sent a hotfix that the support representative said should be installed on every 2008 R2 server whether they experience this issue. This has been a known issue since Server 2008.
Indeed the hotfix is readily available if this has been a known issue for so long. No, it is not. Indeed this major bug was fixed in a product update or service pack for Server 2008 or Server 2008 R2? Not it was not. The hotfix is only available by opening a Microsoft support ticket. The good news is Microsoft fixed the bug in Server 2012.
The bottom line was the customer doubled the number of servers being streamed by PVS and upgraded to IE11, and the new users had large mailboxes and huge Excel files. The NTFS file system could not keep up with the extra demand, and the XenApp 6.5 servers would hard lock when a large Excel file and a large mailbox were opened on the same server.
If you experience this issue, look for A computer that is running Windows Server 2008 R2 SP1 or Windows 7 SP1 freezes under a heavy file I/O load. You know Microsoft, this hotfix is only available from them only if you open a support ticket. I can’t give it to you. 🙂