Windows 2003 Small Business Server Backup fails unexplainably

[feelmjawlk]

Hi!

I’ve been browsing this forum for quite some time but now it’s time to make my first post, a support related one, something that is somewhat regrettable since I would rather start out contributing. Hopefully I will have the chance to give back on a later occasion. Anyways here is my primary cause of the latest head-aches:

Since a couple of weeks I run a Windows Small Business Server at work. It's scheduled to backup the system to a tape drive at night. This was working fine for a couple of weeks until a week ago when the backups started to fail. I have studied the logs of the backup and found that what has happen is that a large number of various files are not backed up. Quotation from log:

/.../

Warning: Unable to open "C:\ClientApps\wxpsp2\i386\WIN9XMIG\MODEMS" - skipped.

Reason: Access is denied.

Warning: Unable to open "C:\Documents and Settings\Administrator" - skipped.

Reason: Access is denied.

Warning: Unable to open "C:\Documents and Settings\Administrator\Start Menu\Programs\Accessories" - skipped.

Reason: Access is denied.

/.../

Since the administrator's home directory was one of the folders that failed to be backed up I took a closer look at the permissions for the folders and files that was being skipped. In order to see if this problem is permission related ("Access is denied" certainly implies that the problem is permission/owner related) I opened up the user properties for Active Directory user MyBusiness/Users/SBSUsers and selected "Member of" and added "Administrators". After last nights run I'm able to conclude that this made no apparent change what so ever.

Taking a closer look at the individual files, for example these two:

Warning: Unable to open "E:\Winner\Winner\BIN\phl_045.dll" - skipped.

Reason: Access is denied.

Warning: Unable to open "E:\Winner\Winner\BIN\phl_046.dll" - skipped.

Reason: Access is denied.

I find that they have the folowing properties:

Security, Group or user name:

Administrators: Full control

ao@ostra.local: Full control

lg@ostra.local: Full control

SYSTEM: Full control

Users: Read & Execute + Read

owner: ao@ostra.local

But in the "bin" directory there is 91 files and only these two failed to be backed up! for example: phl_044.dll and phl_047.dll which have the exact same properties!

I couldn't distinguish any difference between the different files so there for I look at the log from another date:

On that occasion three files in the winner\bin directory was skipped:

Warning: Unable to open "E:\Winner\Winner\BIN\phl_351.dll" - skipped.

Reason: Access is denied.

Warning: Unable to open "E:\Winner\Winner\BIN\phl_358.dll" - skipped.

Reason: Access is denied.

Warning: Unable to open "E:\Winner\Winner\BIN\PHUTIL.DLL" - skipped.

Reason: Access is denied.

Yet again the have the same properties as their neighbouring files. The only pattern I can see is that they come next after each other when listing the directory after name.

This led me to believe that perhaps the problem is caused by third party software? The only third party software I've installed on the machine is:

F-Secure anti-virus for Windows servers

F-Secure Management Agent

F-Secure Policy Manager Console

F-Secure Policy Manager Update Server & Agent

APC PowerChute Business Edition Agent

Since the problems started after the 20:th October I made a search for all files modified on that particular date. From studying the search result I can see that nothing special happened that day. Only some various log files were created / modified and some user created files were uploaded.

Next step is checking the event logs for the first time the backup failed (early morning 21/10):

Event Type: Error

Event Source: NTBackup

Event Category: None

Event ID: 8001

Date: 2005-10-21

Time: 00:42:26

User: N/A

Computer: SERVER

Description:

End Backup of 'C:' 'Warnings or errors were encountered.'

Verify: On

Mode: Replace

Type: Normal

Consult the backup report for more details.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

This error is displayed for drive d: and e: as well. The only object to be successfully backed up was "SERVER\Microsoft Information Store\First Storage Group"

Even the "system state backup" failed.

During the day of the 20:th I find this event that might be interesting:

Event Type: Information

Event Source: SceCli

Event Category: None

Event ID: 1704

Date: 2005-10-20

Time: 13:58:09

User: N/A

Computer: SERVER

Description:

Security policy in the Group policy objects has been applied successfully.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Looking closer to the events during the night when the backup failed for the first time (early morning 21:st)

/…/

Event Type: Warning

Event Source: Userenv

Event Category: None

Event ID: 1524

Date: 2005-10-21

Time: 02:44:38

User: ostra\SBS Backup User

Computer: SERVER

Description:

Windows cannot unload your classes registry file - it is still in use by other applications or services. The file will be unloaded when it is no longer in use.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

/…/

After looking after other occasions the event may have occurred I found that the same event occurred on the 18:th and 12:th and several other days when backup was successful.

The same thing goes for the SceCli event.

So now I'm completely, utterly, inevitably stuck with no more ideas on solving the problem other than reinstalling the whole system, something that I rather not do.

There for I would be more than happy if anybody have any ideas where to look for possible causes to the problem. I have attached some of the recent logs from the backup if anyone have an idea what to look for.

success.txt

fail2.txt

fail1.txt

Edited October 27, 2005 by feelmjawlk

[cluberti]

The Userenv errors are actually very informational, but probably not related. However, it would be wise to install uphclean from Microsoft to clear those up. As to the backup jobs failing, have you gone into the scheduled tasks applet and tried to use a different account for backing up files in the properties of your backup job? It sounds like perhaps you've got a permissions issue - also, make sure that you haven't messed up the bypass traverse checking right in your GPO, or this sort of thing can happen too. The everyone group (or a similarly more secure group, like Authenticated Users) should have permissions here.

[feelmjawlk]

Thank you cluberti for your reply!

Currently the backup application is run as "NT AUTHORITY\SYSTEM". I've also started the backup application manually, logged in as the administrator with the same result. Perhaps I could try to use another user for the task. How would I refer to: "ostra.local\MyBusiness\Users\SBSUsers\Backup User" in the "run as"-box?

If the problem is permission related, why are some files failing during some runs, and during another it's other files (although it seems that the files come from the same directory).

I've looked around in the group policy management but I have made no changes here. I'll look into what the "bypass traverse" is for someting. Haven't heard of it

EDIT:

Allright, I've just checked up on what bypass traverse setting is. Bypass traverse checking for the server in question includes:

Administrators

Authenticated Users

Everyone

Pre-Windows 2000 Compatible Access

Is system included? I know "Backup User" is a member of "Domain Admins" which is a member of "Administrators".

Edited October 28, 2005 by feelmjawlk

[cluberti]

Do you use volume shadow copies on this server? Also, the error is access denied - check one of the files from the list of failed files and make sure that SYSTEM has at least read rights on the file or folder. It would still be wise to use a real user to run the backup job, not SYSTEM - if the username is "backup user" (as you stated), then you simply put "<DOMAIN NAME>\Backup User" in the run as box.

[feelmjawlk]

Thank you for your help cluberti. I've found that it's not ntbackup thats scheduled to run but C:\Program Files\Microsoft Windows Small Business Server\Backup\bkrunner.exe which is run as "system". The permissions doesn't seem to be the problem I've studied them very closely and but can't find any real pattern for what makes a file accessible or not. And yes, backup is done via shadow copy.

Lately I've installed Veritas Backup Exec 10 Demo. It works, but not the shadow copies! So now I'm starting to think that thats where I should look for the solution. I could go with Backup Exec but I would much rather see the SBS Backup progress working.

[tuipveus]

Thank you for your help cluberti. I've found that it's not ntbackup thats scheduled to run but C:\Program Files\Microsoft Windows Small Business Server\Backup\bkrunner.exe which is run as "system". The permissions doesn't seem to be the problem I've studied them very closely and but can't find any real pattern for what makes a file accessible or not. And yes, backup is done via shadow copy.

Lately I've installed Veritas Backup Exec 10 Demo. It works, but not the shadow copies! So now I'm starting to think that thats where I should look for the solution. I could go with Backup Exec but I would much rather see the SBS Backup progress working.

I can confirm that reason for problem is with F-secure. I have exactly same problem with SBS2003. And after I disable f-secure programs from web control, and disable all services backup works perfectly. Windows Shadow copy seems to have random problems with F-secure, or vice versa.