RootUsers

After performing a regular Windows update and restart recently I found that XenServer tools appeared to have been removed. After trying to reinstall the tools I found that the process would hang indefinitely preventing the problem from being resolved.

After contacting Citrix support they advised that this was a known issue caused by Amazon accidentally uploading a set of their WHQL PV tools to Microsoft Windows update which overwrote the existing PV drivers in place, here’s how to fix this.

It’s still not clear how exactly this happened, Citrix have advised that apparently Microsoft allowed Amazon to upload to Windows update on behalf of Citrix which would involve Amazon using Citrix’s vendor ID, it sounds like there may be a security hole in the process somewhere.

Before you begin

• Ensure that you have a snapshot and/or a recent backup of the server, previously I have removed XenServer specific drivers which resulted in a blue screen of death (BSOD). Having a roll back plan is highly recommended before proceeding with these sort of changes.
• Record the network configuration on all network adapters including IP address, Subnet mask, Default gateway and DNS servers. In one instance I removed the network driver and after fixing it the configuration was all gone which is to be expected, I had to enter it back in after fixing the problem. This did not happen on another server that experienced the same issue however, so may not always be the case.

Identify the problem

I first noticed that there was a problem after Windows update and reboot as the virtualization state of the virtual machine showed that XenServer tools were not installed, when they definitely were prior to rebooting.

When I mounted the xs-tools.iso file and attempted to reinstall the tools, the installation would stop on the “Citrix XenServer Tools Installer” screen and showed nothing except for the below for over 24 hours without progressing.

This happened for me with two Windows 2012 R2 Data Center virtual machines running on XenServer 6.2 and 6.5, so it was not XenServer version specific.

At this point I cancelled the installation as it was clearly not working and submitted a support ticket to Citrix, who advised on the following.

Confirming the problem

To confirm that you are facing this particular problem, open device manager and see if you have any of these.

Under System Devices:
– AWS Interface
– AWS PV Bus
– AWS PV Network Class

Under Network Adapters:
– AWS PV Network Device

Under Storage Controllers:
– AWS PV Storage Host Adapter

In my case I had most of these, as shown below.

These AWS drivers confirmed that I was experiencing this known issue. The virtual machine is running on an internally managed Citrix XenServer host and not in AWS so it should not have these AWS specific drivers. These were installed through Windows update as optional updates on the 12th of August 2015.

Fixing the problem

Now that we have confirmed the problem we can manually remove the AWS drivers. Again as noted previously make sure that you have a snapshot in place and/or recent backup that you can roll back to should this process cause any unexpected problems.

Run the following commands in command prompt with administrator privileges.

reg.exe delete HKLM\SYSTEM\CurrentControlSet\Services\XENFILT\UNPLUG /v disks /f
reg.exe delete HKLM\SYSTEM\CurrentControlSet\Services\XENFILT\UNPLUG /v nics /f

Uninstall the drivers

In device manager select view and select show hidden devices. This ensures that all devices are shown so that we don’t miss anything.

In device manager, right click on AWS PV Network Device which can be found under network adapters and select uninstall. In the window that opens, tick the ‘Delete the driver software for this device’ option.

Repeat this process for any remaining AWS PV Network Device, you may have more depending on the amount of network adapters in place.

Repeat this uninstall process for the following AWS devices in this order.

AWS PV Network Class
AWS Interface
AWS PV Storage Host Adapter (you will be prompted to reboot after this, do not do it yet)
AWS PV Bus (you will be prompted to reboot after this, do not do it yet)

In both of my cases I did not have AWS PV Bus present, however you may.

Once these have all been uninstalled, reboot the server from within the guest operating system. You will not be able to perform a reboot through XenCenter as the VM will report as not having the tools installed.

Reinstall XenServer Tools

Now that we have cleared the problematic AWS drivers we can reinstall XenServer tools. Mount the ISO and run the installeizard.msi installer. You may be prompted to remove XenServer tools as they may list as still being installed, if this is the case simply select to remove the tools prior to proceeding.

Once the tools have been removed, run the installer again which should repair the drivers. When the installation has completed reboot when prompted. After the reboot has completed the virtualization state should report that XenServer tools are installed correctly.

Afterwards I ran a check for updates within Windows update and confirmed that the optional updates no longer show, this was a couple of days after the initial problem however so it would appear that they have since been removed.

Still not working?

If this did not resolve the problem, you may need to try the removal again with the installer, and then manually uninstall the following through Programs & Features.

Citrix Xen Windows (x86 or x64) PV Drivers
Citrix XenServer Tools Installer
Citrix XenServer VSS Provider
Citrix XenServer Windows Guest Agent

Be careful when removing these, as this has caused BSOD for me in the past. Once they have been removed, reboot the server and try running the XenServer tools installation again.

Summary

The problem was caused by AWS PV drivers being installed which were listed as optional updates in Windows update. The problem was resolved by manually uninstalling these, removing XenServer tools, reinstalling XenServer tools and then performing a reboot.

I’m not sure how useful this post will be as it is doubtful that this problem will happen again, however hopefully it may help anyone that has the issue that has not yet resolved it.