Can’t Start Hyper-V VMs with Event ID 12140, 12010 and 12030

Had a few Hyper-V host systems today that after rebooting failed to restart the VMs that were set to auto-restart. No updates had been installed – the reboots were due to power environment changes.

Attempting to restart them from Hyper-V Manager simply resulted in the VM status quickly changing from Off to Starting then back to Off.

Digging though the Event Logs (Applications and Service Logs | Microsoft | Windows | Hyper-V-Worker | Admin) resulted in this:

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin
Source:        Microsoft-Windows-Hyper-V-Worker
Date:          4/02/2013 1:42:57 PM
Event ID:      12140
Description:
'hyper-vm1': Failed to open attachment 'E:\hyper-v\VHDs\hyper-vm1.vhd'. Error: 'A device attached to the system is not functioning.' (0x8007001F). (Virtual machine ID 9F3157AA-4875-45C5-BAE4-3D7D5C432B8A)

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin
Source:        Microsoft-Windows-Hyper-V-Worker
Date:          4/02/2013 1:42:57 PM
Event ID:      12010
Description:
'hyper-vm1' Microsoft Emulated IDE Controller (Instance ID {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to Power on with Error 'A device attached to the system is not functioning.' (0x8007001F). (Virtual machine ID 9F3157AA-4875-45C5-BAE4-3D7D5C432B8A)

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin
Source:        Microsoft-Windows-Hyper-V-Worker
Date:          4/02/2013 1:42:57 PM
Event ID:      12030
Description:
'hyper-vm1' failed to start. (Virtual machine ID 9F3157AA-4875-45C5-BAE4-3D7D5C432B8A)

And this one from Hyper-V-VMMS/Admin (Applications and Service Logs | Microsoft | Windows | Hyper-V-VMMS | Admin):

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          4/02/2013 1:37:05 PM
Event ID:      14098
Description:
'Storage Virtualization Service Provider' driver required by the Virtual Machine Management service is not installed or is disabled. Check your settings or try reinstalling the Hyper-V role.

It was this second one that helped me track down the problem. I subsequently found Microsoft Knowledgebase Article 2013544 which listed a similar scenario and recommended changing the FSDepends driver from Manual start to Boot start as follows:

• Start Registry Editor
• Navigate to the following registry key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\FsDepends
• Under the FsDepends key, change REG_DWORD value “Start” from 3 to 0
• Restart the server

The reason is supposedly due to a timing issue between FSDepends.sys (nested volume dependency management driver) and VHDMP.sys (VHD parser and dependency property provider driver), typically triggered by third party backup programs that load tape drivers. This wasn’t the case in my situation, but changing FsDepends from Manual start to Boot start ended up resolving my VM startup problem.

Recovering from WinRM Authentication Lockout

If like me you’re silly enough to lock yourself out of WinRM by removing Kerberos and Negotiate authentication from the WinRM client, you’ll find it a bit difficult to reset the WinRM configuration, because WinRM uses itself to modify the configuration and reset itself (winrm invoke restore).

I wasn’t particularly interested in performing a restore on my laptop, so I went hunting for the registry location for WinRM’s client configuration. The best TechNet could provide me with was “The configuration information is stored in the registry” which is pretty crap, even by Microsoft’s standards.

Resorting to a registry search – thankfully I had added the remote end to the TrustedHosts list – I came up with the registry location:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\Client

Setting auth_kerberos and auth_negotiate to 1

and restarting the Windows Remote Management (WS-Management) service got me up and going again.

Navigating Remote Symlinks on a Windows Server from a Windows Client (or, Poor Man’s DFS Links Without DFS Installed)

I set up a bunch of symlinks in a share on a Windows Server 2008 R2 install, pointing to a range of different UNC paths. My testing on the server showed that the symlink traversal was working fine, but on a Windows 7 install I was getting the following error:

“The symbolic link cannot be followed because its type is disabled.”

Odd error. After much mucking about I found that the fsutil command is used to control this behaviour. The following command was used to display the current symlink evaluation methods:

fsutil behavior query SymlinkEvaluation

which resulted in the following:

Local to local symbolic links are enabled.
Local to remote symbolic links are enabled.
Remote to local symbolic links are disabled.
Remote to remote symbolic links are disabled.

Bingo. The Remote to Local evaluation mode is disabled, which is causing the error. Local to Remote evaluation mode is enabled, which is why the symlink traversal was working on the server. I verified that the problem was resolved by issuing the following command on the Windows 7 install:

fsutil behavior set SymlinkEvaluation L2L:1 L2R:1 R2R:1 R2L:1

Excellent, the symlinks are now followed without error. Finally I rolled out the above change via Group Policy. The four modes can be controlled by using Group Policy Editor and navigating to Computer Configuration > Administrative Templates > System > Filesystem and configuring "Selectively allow the evaluation of a symbolic link".

FreeBSD 8.1, 8,2 and Hyper-V R2 SP1 Install Problem - Use Fixed Size VHDs

Just tried installing FreeBSD 8.1 and 8.2 virtual machines on a Windows Server 2008 R2 Core install with the Hyper-V role installed and with SP1 applied. newfs created the file systems just fine, but the distribution unpacking would cause random kernel panics, throwing ‘ufs_dirbad: bad dir ino XXX at offset XXX: mangled entry’ errors.

I’d created the VHDs as dynamically sized VHDs. I blew these away and created fixed size VHDs and attached them to the VMs. I’ve been repeatedly performing full distribution installs without error. I managed to find a Hyper-V R2 box without SP1 and couldn’t replicate the install problem with dynamically sized VHDs, so Microsoft have introduced a problem with SP1.

If you’re seeing disk-related problems with your UNIX/UNIX-like VMs on Hyper-V, check to see if you’re using dynamically sized VHDs and convert them to fixed size VHDs to see if this fixes the problem.