Tuesday, September 25, 2007

Modifying SBS 2003 SP1's bkprunner.exe for Improved Backup Performance

I'll quickly jot this down before I forget.
I've recently been having a shrinking backup window on one of my client's SBS 2003 boxes. It backs up to tape and I didn't want to create a backup script and lose the nice reporting features that SBS provides. So I hacked the bkprunner.exe process instead :-)
On my own SBS 2003 box I was getting terrible server performance during my daily backup to USB drives. I found the undocumented /FU switch that was included with the SP1 version of ntbackup and some registry modifications that the Exchange team of Microsoft IT performed to improve their backup performance.

Open Explorer and go to "C:\Program Files\Microsoft Windows Small Business Server\Backup"
Make a copy of bkprunner.exe

Download and extract XVI32.
Run XVI32.exe
Open bkprunner.exe in XVI32

The address range $10F0-$11B7 is used for backups to .bkf files
The address range $11B8-$1277 is used for backups to tape

To turn off verify when backing up to a .bkf
Go to address $113A
In the hex pane (the middle one), type in the following hex values:
6E 00 6F 00 20
This enters in the text "no " in Unicode format.

To turn off buffered writes (as explained in MSKB 839272 and also here) when backing up to a .bkf - recommended
Go to address $115E
In the hex pane (the middle one), type in the following hex values:
46 00 55 00 20 00 20 00 20 00 20
This enters in the text "FU " in Unicode format.

To turn off verify when backing up to tape
Go to address $1202
In the hex pane (the middle one), type in the following hex values:
6E 00 6F 00 20
This enters in the text "no " in Unicode format.

Registry modifications for performance
Run regedit
Open HKEY_USERS
Load Hive
Open SBS Backup User's NTUSER.DAT registry hive; call the key name BACKUP
Browse to HKEY_USERS\BACKUP\Software\Microsoft\Ntbackup\Backup Engine.
Edit the value of the entry Logical Disk Buffer Size from 32 to 64.
Edit the value of the entry Max Buffer Size from 512 to 1024.
Edit the value of the entry Max Num Tape Buffers from 9 to 16.
If the above keys don't exist, create them as String values.
Click on HKEY_USERS\BACKUP
Unload hive

35 comments:

Anonymous said...

Chris, nice blog. I added it to my blogroll. :-)

Question - What kind of performance improvement are we talking about here after this change. I'm mainly concerned with B2D, not tape.

Thanks!
-Tim

stryqx said...

Hi Tim,

Thanks for your encouraging words. It's nice to know I'm just not writing some notes for myself.

I'm seeing about a 3-10% reduction in wall clock time for backups, but the big benefit from using the /FU switch is responsiveness of the server.

Prior to using the /FU switch, working with server resources during a backup to .bkf was excruciating. The /FU has made the server useful again during the backup period. There's still a performance hit, but nowhere near the impact of not using /FU.

Anonymous said...

Hi Chris

Some great detective work there :)

Would u know how to change the backup method from full to differential when backing up to disk?

I currently have a customer backing up 100GB of data every night and it's taking about 12hours! I really want to minimise the load on the server.

Cheers,
Darryl

stryqx said...

The issue with going from normal to differential backup would be what gets sacrificed to include the additional characters. Also, you would need to be mindful of the requirements and impacts that differential backups have on both Exchange Server and SQL Server.

I'd be inclined to have the SBS-ised backup perform a full backup weekly, then have a second Scheduled Task run the differential backups. But I'd certainly look at the impact this has on Exchange and SQL if they're included in the differentials. The downside is that this second Scheduled Task won't be integrated into the SBS reporting. You would need to do this yourself with some scripts - Windows IT Pro InstantDoc #44990 would be a good starting point.

Anonymous said...

Chris,

Do you know if your bkprunner modification can be done a SB2003 SP2 R2 server?, I noticed your blog only indicated Service Pack 1.

Thanks

Justin

stryqx said...

@justin - you can certainly do this on an SBS 2003 R2 SP2 box. I'm currently doing this.

For reference, the version number of bkprunner.exe that I've modified is 5.2.2893.0. Let me know if you're running something different.

Anonymous said...

Yeah, you really did good stuff.I've been using workarounds a lot at my customers, but your tip is the golden shot. I'd improves backup performace up to 40% in a few cases, thanks a lot.I wonder why I didn't stumble onto this /FU switch searching MS support pages, may be I'm getting to old ?
Greetings from lower bavaria, Germany, Christian Birr

Ben said...

thanks for the post, good details.

I am running SBS Sp2 and have followed the steps to the letter: BUT the digital signature of bkprunner.exe is no longer present, so now it does not run. *sigh* I added the exe as an exclusion to the DEP, but no luck.
Any suggestions? Or does anyone have a working bkprunner.exe file they can send me?

Cheers,

Ben

stryqx said...

@ben: The only setting I can think that would be causing you problems is the "Allow software to run or install even if the signature is invalid" setting in Internet Explorer. If this is unchecked, then it could be preventing the modified bkprunner.exe from running.
Problem is, I have this unchecked on my SBS SP2 system (for the admin account that the Scheduled Task runs as).
I'm not sure what's preventing yours from running. But your configuration sounds like the correct configuration - modified binaries with digital signatures shouldn't be running. Thanks for the heads-up. I'll now have to investigate why it's running for me.

The IE setting can be found via the following menu hierarchy.

Tools->Internet Options->Advanced->Scroll down to Security->Check the option "Allow software to run or install even if the signature is invalid"

Unknown said...

Hello, Excellent article. I am hoping you may be able to help me solve another issue. I have a customer where we installed a NAS box. It runs a modified Linux kernel and uses SAMBA 3.22. I have joined it to the domain. The strangest thing is it will not back up to the share on this unit using bkprunner.exe. If I run the exact command line used by bkprunner from a prompt it works no problem. It works from a normal ntbackup and we have a schedule of backups running to it but they are not integrated in the monitoring. Any thoughts would be greatly appreciated.

stryqx said...

@US - bkprunner.exe gets launched by a Scheduled Task that runs as the "SBS Backup User" account. Make sure this account has access permissions to your Samba share.

Anonymous said...

Great post! has helped me numerous times, however right now I have one I cannot seem to get an answer for anywhere. SBS2003 SP2 has the /fu flag enabled by default, however for some reason I get this in the backup log:

unable to verify if the target disks satisfy the requirement of the command line file-unbuffered flag (/FU) flag.

The backup then proceeds with buffered write and we all know what that does! Anyway to hack the backuprunner to disable the check for unbuffered write requirements?

stryqx said...

The /FU flag hack in bkprunner gets passed to ntbackup.exe. The /FU flag is then used to set the FILE_FLAG_NO_BUFFERING flag in the CreateFile call when it creates the .bkf file.

The error you're getting is related to either CreateFile or WriteFile returning an error, at which point NTBackup reverts to buffered file access. You'd need to run something like Process Monitor to find out which call was causing the problem.

The File Buffering page on MSDN lists the gory details on the constraints imposed on using this flag - it sounds like you're hitting a buffer alignment problem. What are you backing up to?

Anonymous said...

This is excellent and I have managed to implement this on my SBS2003R2 box.

Is there any way I can get bkprunner working on XP clients as well? My users store a lot of results locally (try telling them not to) and backing up their local data would be agreat help.

stryqx said...

@MarcusN - thanks for the feedback.

Regarding local PC backups: if you've got 10 PCs or less have a look at a Windows Home Server - the Client Backup feature of this product is excellent. You can add this alongside an SBS box, but you can't use the Client Backup program on the SBS box itself. Another way to solve this is to use Group Policy to configure Folder Redirection and Offline Files and Folders. This moves specific folders to the server and makes them available to laptop users when disconnected.

Otherwise you need to look at a synchronisation package that can be scripted.

Steve said...

I have SBS 2003 SP1 and I tried your tips. The problem is that the backup process has not changed. According to the SBS backup log, the backup command line still have verify on and FU not enabled. If you can share any insight on my issue it would be appreciated.

Thanks for the posting. I found it very helpful.

stryqx said...

@steve - are you backing up to tape or to a .bkf file?

I'd open up bkprunner.exe in xvi32 again and verify that the changes are there. Also note that there are two sets of switches - one for backup to tape and the other for backup to .bkf files.

Steve said...

I am backing up to a file. I only employed the changes for the file backup process. I probably have check the file about 5 times by now. I am fairly certain I followed you instructions correctly.

I did notice some 1 area within the hex editor where there was another verify switch. I changed it and will check tonight's scheduled backup.

Thanks. Please let me know if you have another suggestion.

ericwestbo said...

Chris -

Nice blog! Shamefully, you're only one of two sites that even touches on the error reported by DW (12:07 AM, August 28, 2008).

We're running ntbackup on Server 2003 Enterprise & are experiencing the same issue - but only when trying to backup across the firewall. LAN based machines work fine; DMZ machines fall back to buffered write.

Any thoughts? I'm looking at 48+ hour backups on one backup set & its killing our ability to maintain our SLAs.

- KT

stryqx said...

@ericwestbo - without having an environment where I'm getting this error and the ability to debug it, I can only suggest using Firestreamer-RM to see if this improves your backup performance.

Anonymous said...

Fantastic, and yet very simple.

Anonymous said...

I have used your "fix" for a while now but I beleive that the /FU switch is not as useful for decreasing backup times as we think.

My backups just recently started failing verify. So to see where it was failing I did a manual backup at the command line using the full logging switch. When I opened the file right at the top I saw


unable to verify if the target disks satisfy the requirement of the command line file-unbuffered flag (/FU) flag.


I had never noticed that because the SBS 2003 backup script only gives a brief log file.

I am backup up to a Lacie Ethernet disk

I went to here

http://msdn.microsoft.com/en-us/library/cc644950(VS.85).aspx

And read about the alignment needs and it seems to me that the majority of the time those alignment needs will never be met. So the /FU switch will NEVER decrease backup times. BUT it still does INCREASE server responsiveness somehow. This is all backed up by what you yourself said:

I'm seeing about a 3-10% reduction in wall clock time for backups, but the big benefit from using the /FU switch is responsiveness of the server.

It would be nice if you could do some more detailed testing to test this theory.

Anthony Maw said...

The instructions could use a little more detail:

To easily find a hex address in XVI32 hex editor, click "Address/Goto" and enter the desired hex address value.

Then to change a value, click "Edit/Overwrite String..." and copy and paste the values described in this article. Don't forget to hit SAVE.

If you did it right, the icon should not change.
Regards, Anthony Maw, Vancouver, Canada www.anthonymaw.com

Anthony Maw said...

OK well I successfully modified the backuprunner.exe file but the backup logs state: "Warning: unable to verify if the target disks satisfy the requirement of the command line file-unbuffered flag (/FU) flag.
An error code of 0x5 was returned, thus the flag is ignored.
The backup will proceed with buffered write." Does anybody know how to get the /fu switch to work with NTFS formatted USB external hard drives (which is presumably what most people use?)

Anonymous said...

You just saved me 10 hours. my backup went from 16hrs to 6. Thank you!

Josh

marcelo sauaf said...

@ dev.mgr when external USB HD:
1) set HUB controller props. to NOT power off devices to save energy
2) make sure you have the external HD pluged into the ENHANCED USB controller (see @ dev.mgr)
3) set HD policy (props.) as optimized for speed

Regards

Anonymous said...

What hex values would I need to change so the logging switch gets changed from simple to full?
want to change the switch from /l:s to /l:f

stryqx said...

@anonymous - sorry for the long delay in posting this:

To enable full logging when backing up to a .bkf
Go to address $1196
In the hex pane, type in the following hex value:
66
This changes /l:s to /l:f

Bradford said...

Hi Just wanted to say thanks for this. It helped us a lot with our backups so kudos to you sir!

Unknown said...

I had to post and say this went from bkprunner killing the backup at 22hours to it completing in 7. Amazing. Two Questions: if we tweak the backup settings in the server manager, do we have to redo these steps? Once this file is tweaked, can we copy it to other servers?

Bob Dorrestein said...

I tried your suggestion, but still verification was executed. Then I digged deeper into the code of bkprunner.exe and found a "/v: yes" on HEX address 1072. Changing this to "6E 00 6F 00 20" solved my problem. Many thanks for your suggestion.

James said...

This article is amazing. My SBS Console now shows the old green check next to backups and the process is a few hours shorter for 350GB.

Anonymous said...

Thank you. I want these types of improvements too.
Can you please expand the registry editing steps? I don't seem to have a NTUSER.DAT file in my /Documents and Settings/SBS Backup User directory.
>> Run regedit
Open HKEY_USERS
Load Hive
Open SBS Backup User's NTUSER.DAT registry hive; call the key name BACKUP
Browse to HKEY_USERS\BACKUP\Software\Microsoft\Ntbackup\Backup Engine. <<

Thanks, Ken

stryqx said...

Hi Ken,

You'll need to enable "Show hidden files and folders" and maybe also disable "Hide protected operating system files" in Windows Explorer Folder Options if you want to see NTUSER.DAT. You can also just browse to the location and type in NTUSER.DAT to load it.

stryqx said...

Hi Ken,

The changes made to HKEY_CURRENT_USER only make a difference when NTBackup is run as the currently logged on user. The scheduled backup task created by the SBS console runs as the SBS Backup User, which is why we go to the effort of loading that account's registry hive and make the changes there. If you do run NTBackup as the currently logged on user, then by all means make the changes to HKEY_CURRENT_USER as well.