Fixing the Piece of Sh*t that is OneDrive on Azure Virtual Desktop when the Profile Disk Inevitably Corrupts

Profile Disks with FSLogix and Cloud Cache-only (due to lack of Azure AD-only support for Azure Files) can be pretty crappy, given FSLogix provides no pre-mount or post-mount offline chkdsk to ensure file system integrity of the profile volume. This lack of integrity checking invariably breaks things, such as OneDrive sync, which relies on a clean file system. Performing the offline chkdsk is made harder due to the fact the FSLogix VHDX won't mount using Powershell/Disk Manager/diskpart, so the only option is to use an AVD session host to log in to the impacted user, and performing a force-unmount chkdsk of the profile disk that's loaded for the impacted user.

Sometimes the profile is so borked it won't mount. The only option at this point is to look in C:\ProgramData\Fslogix\Cache for the impacted user, then copy an older .VHDX and .meta file for the impacted user back into the blob storage container used for the cloud cache (can check all the session hosts for different versions), or recreate the profile by renaming/deleting the blob storage container used by the impacted user.

TODO: weekly/monthly copy of the cached profiles to a separate Azure VM/storage disk, as this is easier than implementing a backup system for page blobs, given Microsoft can't be bothered providing a backup/restore framework for this.

Have the impacted user log into an AVD session host

Create a text file named '<user>.txt' in the user's profile folder

Make sure all apps are closed (browsers, Office, OneDrive)

Don't bother with ondrive.exe /reset - it rarely fixes the problem, as it doesn't do a proper reset

Log on to the AVD session host as an admin user

From an elevated Command Prompt, run 'mountvol'

You'll need to iterate over the entries listed as NO MOUNT POINTS

dir \\?\Volume{GUID}\\Profile

^ - if it's a profile disk, the first line with show 'Volume in drive \\?\Volume{GUID} is Profile-<user>', and the profile folder contents should list the '<user>.txt' file created earlier

Once you've identified the impacted user profile, run 'chkdsk /f \\?\Volume{GUID}'

^ - Answer Y to a force dismount

Chkdsk will repair the NTFS volume, fixing the underlying problem preventing OneDrive to sync and also fixing up any Office-related file system corruption

Sign out and sign back in to the impacted user on an AVD session host

Quit OneDrive and any other Office apps

Control Panel > User Accounts > Credential Manager > Windows Credentials

- Remove any Microsoft_OneDrive entries and Office entries

- Only virtualapp/didlogical needs to be there

Use Task Manager to end explorer.exe for the impacted user

Run Regedit from Task Manager > File > Run New Task

- Rename HKEY_CURRENT_USER\SOFTWARE\Microsoft\OneDrive to OneDrive.OLD

- Go to HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Desktop\NameSpace and delete the GUID entries for the OneDrive tenant entries ("OneDrive - <M365 name>" and "<M365 name>")

Run CMD from Task Manager > File > Run New Task

- Cd c:\Users\<user>

- Ren "<M365 name>" "<M365 name>.OLD"

- Ren "OneDrive - <M365 name>" "OneDrive - <M365 name>.OLD"

- ^ - gets the old OneDrive folders out of the way

- Cd AppData

- ren Local\OneDrive OneDrive.OLD

- ^ gets rid of old cache + settings

Run explorer from Task Manager > File > Run New Task

The desktop shell will come back up, and now you can re-run OneDrive, sign in as the user and re-sync user content and sync document libraries.

Fixing Exchange Admin Center > Mail Flow > Rules in Incognito Mode

Yet another example of Microsoft's inability to provide a polished implementation of, well, anything.

To overcome the HTTP 500 Something Went Wrong frame you get when trying to access Exchange Admin Center > Mail Flow > Rules in Incognito Mode, Microsoft helpfully mention that this is a problem in a yellow info banner and to fix this enable third-party cookies. Yep, all of them. Hmm, no thanks.

To save you some time, here's the cookies you need to add to the allow list: 

 chrome://settings/cookies

Sites that can always use cookies:

[*.]microsoft.com

[*.]microsoftonline.com

[*.]office.com

outlook.office365.com

You'll still get the yellow info banner, but at least the Rules will now be displayed.

Fixing DISM /restorehealth Issues on Windows 10, Server 2016 / 2019

My notes on repairing broken systems that no longer install updates.

 When /restorehealth fails, check the following:

Perform chkdsk /f on Boot volume

Clear \Windows\SoftwareDistribution\Download after stopping Windows Update service

Set UseWUServer to 0 in HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU if WSUS is being used

Perform DISM /startcomponentcleanup to see if superseded component is causing the problem

Perform DISM /restorehealth to see if problem is fixed

Examine C:\WINDOWS\Logs\DISM\dism.log for errors

Examine C:\WINDOWS\Logs\CBS\CBS.log for errors, especially missing Catalogs

Perform Google Search for associated KB Articles/Updates for missing catalogs

Download update from Microsoft Update Catalog site

Unpack the update with the following command:

Expand <update>.msu -f:* c:\temp

Add the update package to the SxS store with the following command:

Dism /online /add-package /packagepath=c:\temp\<update>.cab

If the missing catalogs are no longer present, or are horribly broken, remove from registry

- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages

- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\PackageDetect

- Backup/export keys first!

- Search for package as well; search on the package listed in CBS.log as 'CBS Catalog missing'

- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\ComponentDetect

- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\PackageIndex

- The reference link for the CBS registry edits -  Windows 10 V1703: Fix for DISM error 0x800F081F

STOP 0x74 on Windows Virtual Machines

 So I've been running into this a fair bit of late.

After chasing down the rabbit hole on the first one, worrying about virtual disk corruption, malware, legacy drivers (it was a P2V image), underlying host physical memory problems, the resolution was surprisingly simple - increase the virtual machine startup memory by 1GB.

Turns out the boot-time drivers had increased in number/size (looking at you anti-malware drivers) and the dynamic memory boot-time driver doesn't kick in early enough to allow the SYSTEM registry hive to be loaded properly.

So if you run across this in a virtual setting check the VM's startup memory size and bump it up.

RDP to Windows 7/8.1/10 - "An Internal Error Has Occurred" or Black Screen Upon Connection

I've had this error pop up from time to time. Most of the time turning off Persistent Bitmap Caching fixes the problem.

It can also fix the problem where you initiate an RDP connection but are presented with a black screen but a fully functional RDP session (confirmed if you use an RMM tool/VNC on the server to shadow the RDP session).

Or for those of you who like to edit saved .RDP files in Notepad:

bitmapcachepersistenable:i:0

Windows Server Essentials 2016 In-Place Upgrade

If anyone else is dumb enough like me to in-place upgrade their 2012 R2 Essentials install to 2016 Essentials, then there's some extra steps you'll need to take:

• After the first reboot, log in, try to open the Dashboard, then leave the login for 30-45 minutes. This should be long enough for .NET Framework to compile the necessary assemblies. Don't try anything else at this point. Reboot.
• Get the latest Cumulative Update installed by downloading it from the Windows Update Catalog. Most of the problems I encountered were due to way too many bugs shipped with the RTM build.
• C:\ProgramData\Microsoft\Windows Server\Logs needs Modify access for the NetworkService account
• You may need to reinstall the Essentials Connector for all clients, but especially for an On Premises Exchange Server. You'll also want to disable then enable the Exchange Server Integration.
• You'll need to make a backup of HKLM\SOFTWARE\Microsoft\Windows Server
• You'll then need to change the assemblies version from 6.3.0.0 to 10.0.0.0 under this key (i.e. look for Version=6.3.0.0 and change to Version=10.0.0.0)
• You'll need to restore the Disabled Tasks under Microsoft\Windows\Windows Server Essentials
• Backup Cleanup, Consistency Check, Macintosh Status Check and Save CEIP Data all have invalid Triggers and Actions
• You'll need to get these settings from a clean 2016 Essentials install

Set Up a Sophos Access Point on a Sophos Firewall in a Different Subnet

My first experience with a Sophos Access Point was a painful affair, as the Access Point (an AP55) was on a subnet sitting behind a separate router and the subnet’s DHCP server wasn’t the XG Firewall.

After way too much mucking around I finally came across the following article:
How to troubleshoot registration issues for the Sophos Access Point

The upshot was to add Option 234 to the DHCP scope, with Option 234 pointing to the IP address of the XG Firewall you want the Access Point to register with.

For completeness I also placed the Access Point onto the same subnet as the XG Firewall and it also failed to register with the XG Firewall, as the DHCP server for the subnet was a Windows server. After adding Option 234 to the DHCP scope the AP55 showed up on the XG Firewall.

Throttle WSUS Bandwidth During Business Hours

I keep forgetting the appcmd syntax to set/unset maxBandwidth so I can throttle overall WSUS downloads. This gives me the benefit of reducing link congestion during business hours, and providing maximum link utilization out of hours.

rem Apply Bandwidth throttle to WSUS Administration site
%SYSTEMROOT%\System32\inetsrv\appcmd.exe set config -section:system.applicationHost/sites "/[name='WSUS Administration'].limits.maxBandwidth:76800" /commit:apphost

rem Remove Bandwidth throttle from WSUS Administration site
%SYSTEMROOT%\System32\inetsrv\appcmd.exe clear config -section:system.applicationHost/sites "/[name='WSUS Administration'].limits" /commit:apphost

Pop the first one into a script and use Task Scheduler to turn it on prior to business hours. Pop the second one into a script and use Task Scheduler to turn it off after business hours. You could of course use the first script with a higher maxBandwidth setting if you want out of hours to still be throttled, just not so much as business hours.

Creating a Bootable Mac OS X Mavericks ISO

I'm never going to remember this next time I have to do it - saving it for posterity (and the ability for me to find it easily).
The original reference is found at http://forums.appleinsider.com/t/159955/howto-create-bootable-mavericks-iso#post_2412005. Thanks CrEOF!

# Mount the installer image
hdiutil attach /Applications/Install\ OS\ X\ Mavericks.app/Contents/SharedSupport/InstallESD.dmg -noverify -nobrowse -mountpoint /Volumes/install_app

# Convert the boot image to a sparse bundle
hdiutil convert /Volumes/install_app/BaseSystem.dmg -format UDSP -o /tmp/Mavericks

# Increase the sparse bundle capacity to accommodate the packages
hdiutil resize -size 8g /tmp/Mavericks.sparseimage

# Mount the sparse bundle for package addition hdiutil attach /tmp/Mavericks.sparseimage -noverify -nobrowse -mountpoint /Volumes/install_build

# Remove Package link and replace with actual files
rm /Volumes/install_build/System/Installation/Packages
cp -rp /Volumes/install_app/Packages /Volumes/install_build/System/Installation/

# Unmount the installer image
hdiutil detach /Volumes/install_app

# Unmount the sparse bundle
hdiutil detach /Volumes/install_build

# Resize the partition in the sparse bundle to remove any free space
hdiutil resize -size `hdiutil resize -limits /tmp/Mavericks.sparseimage | tail -n 1 | awk '{ print $1 }'`b /tmp/Mavericks.sparseimage

# Convert the sparse bundle to ISO/CD master hdiutil convert /tmp/Mavericks.sparseimage -format UDTO -o /tmp/Mavericks

# Remove the sparse bundle
rm /tmp/Mavericks.sparseimage

# Rename the ISO and move it to the desktop 
mv /tmp/Mavericks.cdr ~/Desktop/Mavericks.iso

Reset 30 Day Grace Timer for Windows XP/Windows Server 2003

I had to do this today as part of recovering a system from hardware failure. This won't be the last time I'll have to do this, so documented for the next time...

rundll32.exe syssetup,SetupOobeBnk

iOS 3CXPhone Settings for Internode

This is the 3CXPhone profile settings on my wife’s iPhone 4S to connect to her Internode NodePhone account (so I don’t forget it for next time):

• Name: Internode
• Display: Internode
• Username: <NodePhone number>
• ID: <NodePhone number>
• Password: <NodePhone password>
• Internal PBX Address: 203.2.134.1
• External PBX Address: 203.2.134.1

Microsoft Online Speed Test Alternative

I keep forgetting this, but the old Microsoft Online Speed Test tool that was at http://speedtest.microsoftonline.com/ is no longer active.
The alternative tool to use is the Office365 Lync Online Transport Reliability IP Probe (TRIPP) tool, located here:

• Amsterdam, NL: http://trippams.online.lync.com
• Blue Ridge, VA: http://trippbl2.online.lync.com
• Dublin, IE: http://trippdb3.online.lync.com
• Hong Kong: http://tripphkn.online.lync.com
• San Antonio, TX: http://trippsn2.online.lync.com
• Singapore: http://trippsg1.online.lync.com

This tool performs the same set of tests that the now defunct Speed Test tool did. Oh, and Java is required to run the tests, so make sure your Java install is up-to-date with the Web plugin enabled.

Can’t Start Hyper-V VMs with Event ID 12140, 12010 and 12030

Had a few Hyper-V host systems today that after rebooting failed to restart the VMs that were set to auto-restart. No updates had been installed – the reboots were due to power environment changes.

Attempting to restart them from Hyper-V Manager simply resulted in the VM status quickly changing from Off to Starting then back to Off.

Digging though the Event Logs (Applications and Service Logs | Microsoft | Windows | Hyper-V-Worker | Admin) resulted in this:

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin
Source:        Microsoft-Windows-Hyper-V-Worker
Date:          4/02/2013 1:42:57 PM
Event ID:      12140
Description:
'hyper-vm1': Failed to open attachment 'E:\hyper-v\VHDs\hyper-vm1.vhd'. Error: 'A device attached to the system is not functioning.' (0x8007001F). (Virtual machine ID 9F3157AA-4875-45C5-BAE4-3D7D5C432B8A)

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin
Source:        Microsoft-Windows-Hyper-V-Worker
Date:          4/02/2013 1:42:57 PM
Event ID:      12010
Description:
'hyper-vm1' Microsoft Emulated IDE Controller (Instance ID {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to Power on with Error 'A device attached to the system is not functioning.' (0x8007001F). (Virtual machine ID 9F3157AA-4875-45C5-BAE4-3D7D5C432B8A)

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin
Source:        Microsoft-Windows-Hyper-V-Worker
Date:          4/02/2013 1:42:57 PM
Event ID:      12030
Description:
'hyper-vm1' failed to start. (Virtual machine ID 9F3157AA-4875-45C5-BAE4-3D7D5C432B8A)

And this one from Hyper-V-VMMS/Admin (Applications and Service Logs | Microsoft | Windows | Hyper-V-VMMS | Admin):

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Date:          4/02/2013 1:37:05 PM
Event ID:      14098
Description:
'Storage Virtualization Service Provider' driver required by the Virtual Machine Management service is not installed or is disabled. Check your settings or try reinstalling the Hyper-V role.

It was this second one that helped me track down the problem. I subsequently found Microsoft Knowledgebase Article 2013544 which listed a similar scenario and recommended changing the FSDepends driver from Manual start to Boot start as follows:

• Start Registry Editor
• Navigate to the following registry key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\FsDepends
• Under the FsDepends key, change REG_DWORD value “Start” from 3 to 0
• Restart the server

The reason is supposedly due to a timing issue between FSDepends.sys (nested volume dependency management driver) and VHDMP.sys (VHD parser and dependency property provider driver), typically triggered by third party backup programs that load tape drivers. This wasn’t the case in my situation, but changing FsDepends from Manual start to Boot start ended up resolving my VM startup problem.

Installing .NET Framework 3.5 on Windows Server 2012 and Windows 8

If you're getting an installation error code of 0x800F0906 while trying to install .NET Framework 3.5 on a Windows 8 or Windows Server 2012 system, it's because the initial installation source isn't available and you're most likely using WSUS without an appropriate Group Policy Object to redirect to an alternate installation path.

There are a few ways of handling this - use the installation media and DISM to install it, set up a GPO to use Windows Update as an alternate installation path, or copy the WinSxS folder off the install media to a network share and configuring a GPO to use this share as an alternate installation path.

If you have the installation media and you need to only do this for a single PC, then the following command will work:

dism.exe /online /enable-feature /featurename:NetFX3 /Source:D:\sources\sxs

You may need to replace D: with the drive letter containing the installation media.

To configure a GPO to use Windows Update, open up Group Policy Management, create and edit a new GPO. Go to Computer Configuration, Policies, Administrative Templates, System. Open up "Specify settings for optional component installation and component repair", change the setting from Not Configured to Enabled and tick "Contact Windows Update directly to download repair content instead of Windows Server Update Services (WSUS)". Click OK, close the Group Policy Management Editor window and link the GPO to an appropriate container in AD, then run gpupdate /force on the affected computer.

If you want to configure a GPO to use a network share, copy the \sources\sxs folder from either a Windows 8 or Windows Server 2012 DVD/ISO to an appropriate location on a server (e.g. \\server\install\win8sxs), then create and edit a new GPO as outlined above. Instead of enabling the WSUS option though, put the network path to the SxS folder in the field for "Alternate source file path".

Microsoft also have a knowledge base article on this here - Error codes when you try to install the .NET Framework 3.5 in Windows 8 or in Windows Server 2012

Microsoft Camera Codec Pack Update for Windows 8 and Windows 8 RT – Woot!

Microsoft have released an update for Windows 8 and Windows 8 RT that provides support for device-specific RAW formats, allowing you to preview these files in Explorer as well as display them in any program that uses the Windows Imaging Codecs.

The Microsoft Camera Codec Pack provides support for the following device formats:

• Canon: Digital Rebel XT, Digital Rebel XTi, EOS 10D, EOS 20D, EOS 30D, EOS 40D, EOS 50D Digital, EOS 60D, EOS 300D, EOS 350D, EOS 400D, EOS 450D, EOS 500D, EOS 550D, EOS 600D, EOS 1000D, EOS 1100D, EOS 5D, EOS 5D Mark II, EOS 5D Mark III, EOS 7D Digital, EOS D30, EOS D60, EOS Digital Rebel, EOS Kiss Digital, EOS Kiss Digital N, EOS Kiss Digital X, EOS Kiss F, EOS Kiss X2, EOS Kiss X3, EOS Kiss X4, EOS Kiss X5, EOS Kiss X50, EOS Rebel T1i, EOS Rebel T2i, EOS Rebel T3, EOS Rebel T3i, EOS Rebel XS, EOS Rebel XSi, EOS-1D, EOS-1D Mark II, EOS-1D Mark II N, EOS-1D Mark III, EOS-1D Mark IV, EOS-1Ds, EOS-1Ds Mark II, EOS-1Ds Mark III, PowerShot G2, PowerShot G3, PowerShot G5, PowerShot G6, PowerShot G9, PowerShot G10, PowerShot G11, PowerShot Pro1, PowerShot S90, PowerShot S95, PowerShot SX1 IS
• Nikon: 1 J1, 1 V1, Coolpix P6000, D1H, D2H, D2Hs, D2X, D2Xs, D3, D3s, D3X, D4, D40, D40x, D50, D60, D70, D70s, D80, D90, D100, D200, D300, D300s, D700, D800, D800E, D3000, D3100, D3200, D5000, D5100, D7000
• Sony: DSLR-A100, DSLR-A200, DSLR-A230, DSLR-A300, DSLR-A330, DSLR-A350, DSLR-A380, DSLR-A500, DSLR-A550, DSLR-A560, DSLR-A580, DSLR-A700, DSLR-A850, DSLR-A900, Alpha NEX-3, Alpha NEX-5, Alpha NEX-5N, Alpha SLT-A55/A55V, Cyber-shot DSC-R1
• Olympus: C-7070 Wide Zoom, C-8080 Wide Zoom, E-1, E-3, E-10, E-20, E-30, E-420, E-450, E-520, E-620, EVOLT E-300, EVOLT E-330, EVOLT E-400, EVOLT E-410, EVOLT E-500, EVOLT E-510, PEN E-P1, PEN E-P2, PEN E-PL1
• Pentax (PEF formats only): *ist D, *ist DL, *ist DS, K10D, K20D, K100D, K100D Super, K110D, K200D, K-5, K-7, K-r, K-x
• Leica: DIGILUX 3, D-LUX 4, M8, M8.2, M9
• Konica Minolta: ALPHA-7 DIGITAL, DiMAGE A1, DiMAGE A2, DYNAX 7D, Maxxum 7D
• Epson: R-D1
• Panasonic: Lumix DMC-G1, Lumix DMC-GH1, Lumix DMC-GF1, Lumix DMC-LX3, Lumix DMC-LX5
• Casio: EX-FH20
• Kodak: EasyShare Z981, EasyShare Z1015 IS
• Samsung: NX11

The update can be downloaded from here:

An update that adds Microsoft Camera Codec Pack support to Windows 8 and Windows RT is available

Hyper-V Integration Components for FreeBSD – Patchfiles

Call me old fashioned, but I’d much prefer a patchset than having to install a version control package and suck down a source code check out. So please find a patchset for the Hyper-V integration components for the following versions of FreeBSD:

FreeBSD 8.2 Hyper-V Integration Components Patchset

FreeBSD 8.3 Hyper-V Integration Components Patchset

FreeBSD 9.0 Hyper-V Integration Components Patchset

FreeBSD 9.1-BETA1 Hyper-V Integration Components Patchset

Download the patchset, then issue:

patch –p –d /usr/src < <patchsetfile>

to patch the source tree, followed by:

cd /usr/src; make kernel KERNCONF=HYPERV_VM INSTKERNNAME=kernel.HYPERV

to install the Hyper-V enabled kernel to /boot/kernel.HYPERV.

Before booting to the Hyper-V enabled kernel it’s best to use GEOM labels to mount the partitions. Follow the instructions here to do this. This makes it easy for you to quickly swap between a Hyper-V enabled kernel and a non-Hyper-V enabled kernel – the reason being the Fast IDE storage driver presents itself as a SCSI driver, changing the device node path which prevents /etc/fstab from working correctly.

It’s worth noting that although I’ve fixed the modules from compiling (compared with the git clone source I pulled down), loading them from a non-Hyper-V enabled kernel will cause a kernel panic. So you need the integration components compiled into the kernel via the HYPERV kernel option.

The other problem I’ve found is that the network driver mostly works for UDP traffic, but regularly stalls on TCP traffic. Hadn’t had a chance to debug it yet.

Very happy with the increased disk performance, the ability to get heartbeat information and the ability to cleanly shut down the guests from the Hyper-V host. Looking forward to KVP communication and a working network driver.

Hyper-V Integration Components for FreeBSD 8.2 has landed!

The Microsoft Openness Blog has just announced that the github repository for FreeBSD 8.2 Hyper-V integration components is now live! This is currently a public beta for evaluation purposes only, so expect some rough edges still.

Instructions for compiling the source code and installing the drivers can be found here. There’s also a mailing list for suggestions and code improvement.

This gives us heartbeat, time sync, shutdown and accelerated network, IDE and SCSI drivers for FreeBSD 8.2 on Hyper-V Server 2008 R2 and Windows Server 2008 R2 with the Hyper-V role. It’s a pity that this won’t land in time for inclusion into FreeBSD 9.1, but it would be good to see it hit –current and –stable in time for any subsequent releases.

Guess what I’m doing over the weekend? :-)

Agentless Bandwidth Testing on Windows

I needed BWping and HTTPing running on Windows for bandwidth and latency testing of some 3G WAN tails so I compiled them using Cygwin. They can be found here and here respectively.

I find QCheck to be a nice tool for bandwidth testing on Windows systems, but it does require a Windows system either side of the link you’re testing.

Null Routes on Windows 7

Null routes are a useful way to quickly discard packets from an unwanted address or network, especially when you’ve not got immediate or any access to the upstream/gateway router.

I had a client PC that was being hammered over a port forward from a router I had no administrative control. I logged a support request for the upstream router, but rather than wait two days to chase up the request, I added a null route to the client PC.

Typically I add a route to a non-existent IP on the network, but the upstream router was intercepting the ARP requests for the non-existent IP and forwarding on the packet.

I then tried adding a route for the host to point to the loopback address (127.0.0.1), but got a “The route addition failed: The parameter is incorrect” error. Helpful.

After trial and error I got the null route working by specifying the current default gateway address and the software loopback interface like this:

route  -p add <IP address> mask 255.255.255.2555 <gateway address> if 1

You may need to use route print to check to see that the interface number for the loopback interface is 1. If the number isn’t 1, then use that number instead of 1 above.

If you’re looking at null routing for sshd/OpenSSH/RDP, then have a look at the ServerFault entries here and here.

Recovering from WinRM Authentication Lockout

If like me you’re silly enough to lock yourself out of WinRM by removing Kerberos and Negotiate authentication from the WinRM client, you’ll find it a bit difficult to reset the WinRM configuration, because WinRM uses itself to modify the configuration and reset itself (winrm invoke restore).

I wasn’t particularly interested in performing a restore on my laptop, so I went hunting for the registry location for WinRM’s client configuration. The best TechNet could provide me with was “The configuration information is stored in the registry” which is pretty crap, even by Microsoft’s standards.

Resorting to a registry search – thankfully I had added the remote end to the TrustedHosts list – I came up with the registry location:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\Client

Setting auth_kerberos and auth_negotiate to 1

and restarting the Windows Remote Management (WS-Management) service got me up and going again.

Useful Network Connectivity Tool

Off the back of my previous Windows Server Developer Preview problem I also came across the Microsoft Internet Connectivity Evaluation Tool. Quite useful for determining the NAT capability, ECN capability, TCP throughput, UPnP capability and multiple connection capability of your router.

Windows Server 8 Developer Preview - Networking Problem

So I fired up a copy on a spare whitebox only to find HTTP and SMB outbound traffic timing out. Bizarrely ping and traceroute were working, so ICMP and UDP were working, as was inbound SMB connections – just not outbound. Did the usual tricks – upgrade network drivers, disabled NIC-based offloading and modified the usual suspects via netsh (Task Offload, Chimney Offload, RWIN tuning) to no avail.

It this point I compared the ‘netsh int tcp show global’ and ‘netsh int ip show global’ outputs with the defaults from a Windows Server 2008 R2 box and noticed that ‘ECN Capability’ in the TCP Global Parameters for Windows Server 8 Developer Preview was Enabled. I set this to disabled using:

netsh int tcp set global ecn=disabled

and outbound connectivity was established.

Images Are Fixed Now

I've restored all the blog images now. Did I mention I hate Picasa Albums?

Broken Images on Blog :-(

I've broken all but one image reference on the blog :-( Please be patient with me while I resurrect them - somehow Windows Live Writer can open the blog post entries with the images intact. Frakking Picasa albums!

Windows DHCP Server – MMC Console Icons Reference

The DHCP Server MMC Snap-in annoyingly doesn’t have a legend for what the icons mean. Every time I debug a DHCP Server-related issue I’ve forgotten what the icons mean from the last time I’ve done it.

Here are the references up on TechNet:

DHCP console icons reference - Windows Server 2003, 2008

DHCP console icons reference - updated for Win2008 R2

Emulating %LOGONSERVER% For Computer Startup Scripts

%LOGONSERVER% is a useful environment variable to use in logon scripts to see which DC has serviced your request and can be handy to reference if you want to access additional files/shares on the DC. Unfortunately this environment variable is only accessible after logon and isn’t useful for computer startup scripts.

When I need to access the DC that’s providing me with GPOs during a computer startup script I emulate %LOGONSERVER% with the following code:

for /f "tokens=1 delims=\" %%i in ('@echo %0') do set DOMCTLR=\\%%i

%DOMCTLR% can now be used in the same way that
%LOGONSERVER% is used.

Workarounds For When “Add Virtual Hard Disk Wizard” Fails (Which Seems To Be All The Time…)

Adding new fixed sized VHDs using the Add Virtual Hard Disk Wizard in the Hyper-V console for some reason has stopped working for me on just about all my installs, with no errors logged. The VHD is created, but the progress slider bar never progresses and it will sit there, forever. It’s got to the point where I don’t use it and haven’t the time to debug the underlying cause.

A GUI-friendly way to work around this problem is to point Computer Management at the Hyper-V host and use Disk Management to Create the VHD.

Another way of doing this quickly is by using VHD Tool – although this doesn’t zero out the VHD and can leak information from the Hyper-V host and previous virtual machine’s disks into the newly created VHD. It is a great tool for lab work though.

Diskpart can also be used to create the VHD from the command line on the Hyper-V host. The command to do this is:

create vdisk file=”d:\path\to\file.vhd” maximum=<size in MB> type=fixed

UPDATE: So apparently I'm getting this error on networks where the domain controllers are still running Windows Server 2003 and an authoritative restore of Active Directory has been performed. The fix for this is to install MSKB 939820 on all the Windows Server 2003-based domain controllers in the affected domain. Interestingly I only found this trying to resolve a System Center Essentials 2010 installation.

Navigating Remote Symlinks on a Windows Server from a Windows Client (or, Poor Man’s DFS Links Without DFS Installed)

I set up a bunch of symlinks in a share on a Windows Server 2008 R2 install, pointing to a range of different UNC paths. My testing on the server showed that the symlink traversal was working fine, but on a Windows 7 install I was getting the following error:

“The symbolic link cannot be followed because its type is disabled.”

Odd error. After much mucking about I found that the fsutil command is used to control this behaviour. The following command was used to display the current symlink evaluation methods:

fsutil behavior query SymlinkEvaluation

which resulted in the following:

Local to local symbolic links are enabled.
Local to remote symbolic links are enabled.
Remote to local symbolic links are disabled.
Remote to remote symbolic links are disabled.

Bingo. The Remote to Local evaluation mode is disabled, which is causing the error. Local to Remote evaluation mode is enabled, which is why the symlink traversal was working on the server. I verified that the problem was resolved by issuing the following command on the Windows 7 install:

fsutil behavior set SymlinkEvaluation L2L:1 L2R:1 R2R:1 R2L:1

Excellent, the symlinks are now followed without error. Finally I rolled out the above change via Group Policy. The four modes can be controlled by using Group Policy Editor and navigating to Computer Configuration > Administrative Templates > System > Filesystem and configuring "Selectively allow the evaluation of a symbolic link".

Outsourced Authentication – Smart or Dumb?

A couple of months ago I closed my Facebook account, partly because of the continual privacy abuse by Facebook, but mostly because of what I thought was poor tooling for managing my social graph and timeline.

Since that point I’ve noticed more and more companies outsourcing their authentication mechanism to Facebook. Smart or dumb? Smart, because you’ve offloaded a password database that you can’t lose or have compromised, although you still have a client database that can. Dumb, because you’ve lost a prospect or customer like me.

If you’re going to outsource authentication it might be an idea to use OpenID instead. OpenID Explained is a good site to understand how OpenID operates. It’s worth noting that most of the major Web players are already OpenID Providers. If you don’t have an existing account with an OpenID Provider, then MyOpenID is a good place to start.

Office 2010 SP1 Is Death For Access Developers

My talented wife started complaining last week that Microsoft Access started continually crashing trying to open databases after performing some design modification.

Some cursory debugging wasn’t providing consistent bugchecks, so rather than putting more effort into understanding the symptom I then looked for a cause. Design edits were working the week before the crashes so I then looked at updates. Office 2010 SP1 had been installed during that time, so I uninstalled SP1 and tried again. Bingo, database editing no longer resulted in Access crashes.

If Access databases are suddenly crashing on you for no reason, check to see if Office 2010 SP1 is installed.

UPDATE: Microsoft fixed this with a hotfix described in MSKB 2553385.

FreeBSD 8.1, 8,2 and Hyper-V R2 SP1 Install Problem - Use Fixed Size VHDs

Just tried installing FreeBSD 8.1 and 8.2 virtual machines on a Windows Server 2008 R2 Core install with the Hyper-V role installed and with SP1 applied. newfs created the file systems just fine, but the distribution unpacking would cause random kernel panics, throwing ‘ufs_dirbad: bad dir ino XXX at offset XXX: mangled entry’ errors.

I’d created the VHDs as dynamically sized VHDs. I blew these away and created fixed size VHDs and attached them to the VMs. I’ve been repeatedly performing full distribution installs without error. I managed to find a Hyper-V R2 box without SP1 and couldn’t replicate the install problem with dynamically sized VHDs, so Microsoft have introduced a problem with SP1.

If you’re seeing disk-related problems with your UNIX/UNIX-like VMs on Hyper-V, check to see if you’re using dynamically sized VHDs and convert them to fixed size VHDs to see if this fixes the problem.

Running chkdsk on a Drive Allocated to Windows Server Backup

Occasionally I see Windows Server Backup throw odd errors pointing to problems with the disk allocated to Windows Server Backup, such as the bizarre “There is not enough space on the disk” – bizarre in that Windows Server Backup is supposed to automagically manage the disk space allocation and tidy up.

The normal course of action would be to run chkdsk /f on the drive, but the drive doesn’t have a drive letter allocated to it. Nor are you supposed to allocate a drive letter to it. The solution? Use the Volume GUID.

To find the Volume GUID, type in the following at an elevated Command Prompt:

mountvol

This will return the command syntax for the mountvol command, followed by the existing volumes and their mount points. We’re interested in the Volume GUID immediately above this line:

*** NO MOUNT POINTS ***

It will look something like this:

\\?\Volume{12345678-1234-5678-9abc-123456789abc}\

We now take this Volume GUID minus the trailing slash and feed it to chkdsk, like this:

chkdsk /f \\?\Volume{12345678-1234-5678-9abc-123456789abc}

This will then allow chkdsk to perform a consistency check and fix of the drive allocated to Windows Server Backup without needing to allocate a drive letter.

Cannot Install RSAT on Windows 7 with SP1

If you try and install Remote Server Administration Tools for Windows 7 on a Windows 7 PC with SP1 installed, you’ll get the following error: "The update is not applicable to your computer."

Either install RSAT prior to installing SP1 or wait until Remote Server Administration Tools for Windows 7 with SP1 is released in Spring 2011 (March-May for those of us who are Northern Hemisphere challenged).

IPocalypse Now + Resources to Learn IPv6

Happy IPocalypse Day – APNIC today were allocated the two remaining /8 networks from IANA. This means that all the free IP addresses have now been assigned to the various regional registrars and that the free pool of IPv4 addresses will be used up over the coming years (months?), which will make life interesting for hosting businesses. It’s probably a good idea to track the Potaroo blog if you’re interested in global IPv6 developments.

If you've got anything to do with the operations of a computer network or deal with hosting in any way now's the time to start learning about IPv6. Here are some useful links to Web sites and books to learn about IPv6:

Sites:
IPv6 Survival Guide - TechNet Wiki
Microsoft Internet Protocol Version 6 (IPv6) - TechNet
The Lazy Admin - IPv6 101–Part 1
The Lazy Admin - IPv6 101-Part 2
The Lazy Admin - IPv6 101-Part 3
The Lazy Admin - IPv6 101-Part 4

Books:
IPv6 Essentials, Second Edition (Silvia Hagen, O'Reilly Media)
IPv6 Network Administration (Niall Richard Murphy & David Malone, O'Reilly Media)
Understanding IPv6, Second Edition (Joseph Davies, Microsoft Press)

For those of you that dislike anything Microsoft please don't dismiss those links or books. Microsoft to their credit have been very proactive in the deployment and transition of IPv6 and have some excellent IPv6 resources. If you have any good IPv6 resources not listed above, please share them! I’ll update this post accordingly.

Windows Home Server on Hyper-V – Resizing the Partition

Yes I know that Windows Home Server has Drive Extender. Yes I know that Drive Extender makes adding storage space easy and is a brilliant piece of technology. However I wanted my WHS install to have a resilient System disk and besides, I didn’t have a spare box for WHS. So I put it on my server running Hyper-V, but clearly didn’t give it enough disk space.

So here’s the process for adding more disk space to a virtualised WHS install:

• Shut down the WHS virtual machine
• Use the Edit Disk action to increase the capacity of the VHD file used by the WHS virtual machine
• Start up the WHS virtual machine
• Log in to the desktop on the WHS virtual machine
• Run Command Prompt
• Run diskpart
• select disk 0
• list partition
• select partition 2 (assumes that you’re using one VHD file and you want to extend the single data partition to fill the unused disk space)
• extend
• exit

Yes I know you’re not supposed to do this. Yes I know you’re supposed to add additional disks (by adding another VHD file). But this works fine for me. YMMV, so take a backup first!

Compiling Firebird 1.5.x on FreeBSD 4.x Requires GCC 3.2

Note to self: when compiling 5+ year old code on a 5+ year old operating system, it helps to use a version of GCC that compiles the resultant code cleanly and more importantly in a portable manner.

GCC 3.3 has a broken libstdc++ that prevents static linking – you end up with unresolved symbols.

Both GCC 3.3 and GCC 3.4 will end up requiring dynamic linking of libstdc++ and libgcc_s – not helpful if you’re trying to be portable and don’t want to pollute a system with the gcc33 or gcc34 package.

Hopefully I’ve seen the last of this problem, but if I don’t write it down now it will only turn around and bite me in several years time

Resetting MMC User Preferences (Restoring Column Sort Orders)

One of the things that annoys me about MMC is the inability to remove column sort orders. In particular I like the default sort order for the DNS Manager snap-in, but once a column sort order has been applied there is no way inside the MMC console to remove column sort orders to revert to the default unsorted view.

The only way to restore the default view is to remove the customisation file for the MMC snap-in in question.

The MMC snap-in preferences files are located at:

%APPDATA%\Microsoft\MMC

Make sure the MMC snap-in is closed, then rename the snap-in preference file by adding a suffix like “-old”. Restart the MMC snap-in and it will be reset to its default settings.