Server 2003 locks up every few days

[agreenbhm]

We have been experiencing an issue on one of our servers where the system will lock up and not allow any new windows to be opened, however, programs that are already open are still accessible from the console. No one can connect to the server using the MAS application when it is in this state. The only fix we’ve found so far is to reboot the server.

Event Viewer is showing errors 2019 and 333.

2019 states:

Event Type:	Error
Event Source:	Srv
Event Category:	None
Event ID:	2019
Date:		2/10/2010
Time:		8:36:10 AM
User:		N/A
Computer:	TUCKER
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 01 00 54 00   ......T.
0008: 00 00 00 00 e3 07 00 c0   ....ã..À
0010: 00 00 00 00 9a 00 00 c0   ....š..À
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 02 00 00 00			   ....

and error 333 states:

Event Type:	Error
Event Source:	Application Popup
Event Category:	None
Event ID:	333
Date:		2/10/2010
Time:		7:38:18 AM
User:		N/A
Computer:	TUCKER
Description:
An I/O operation initiated by the Registry failed unrecoverably. The Registry could not read in, or write out, or flush, one of the files that contain the system's image of the Registry.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 01 00 6c 00   ......l.
0008: 00 00 00 00 4d 01 00 c0   ....M..À
0010: 00 00 00 00 4d 01 00 c0   ....M..À
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........

I’ve been researching this for a while (it’s been happening sporadically the last few weeks), and so far have not found a solution that works. I’ve tried moving our page file to different volumes, as someone suggested, with no success. This problem seems to be occurring sometime at night. Our backup runs at 9pm daily. The only things I can think of that has changed on this machine is that the backup software was updated to the latest build (we are using Acronis True Image Echo Enterprise Server) and we installed a tax table update. I believe that the updates were done at least a few weeks before the problems started happening. Now this issue occurs once every 3 or 4 days. I’m not sure if it’s related, but it sounds like it could be. What would you suggest looking at? I’ve already examined non-paged memory and handles via task manager when the backup has been running, and I haven’t seen anything out of the ordinary. I should add, this server is running 2GB of RAM (not exactly the newest machine).

Thanks in advance.

Edited February 10, 2010 by agreenbhm

[Tripredacus]

Can you post your server config, including what RAID levels you are using?

[MrJinje]

Wouldn't hurt to run a chkdsk /F.

That will rule out a bad sector on your HDD where you HIVE's are located.

Edited February 10, 2010 by MrJinje

[cluberti]

A 2019 will cause a 333, because nonpaged pool is required to do the map for the disk write. The registry is memory mapped in sections, so when a section needs to be flushed it will fail, but the 2019 indicates you're out of nonpaged pool memory to satisfy requests (and once you hit this, it will *always* fail until the reboot, even if NPP frees up to satisfy a future request). Poolmon will catch this, but it's not always easy to use.

I've got a tool that you can use to catch this, linked at the end of this post.

1. Extract the linked file to C:\Poolmon3
   
2. Double click C:\Poolmon3\_LogPool-as-a-service.cmd to start the actual logging
   

This will install poolmon as a service on the machine (obviously this should be run as an administrator on the machine) via srvany, and generate data in a folder called Poolmon-OUTPUT inside the C:\Poolmon3 folder. Let it run until you start seeing the 2019 events again, and then run the _RemovePoolmon3Service.cmd file to stop the service (and thus the logging) after you've let it capture ~5 minutes of 2019s. Once you've done that, compress the Poolmon-OUTPUT folder and attach it to a new post here, and we can tell you what's causing the 2019s (also, feel free to reboot the server once you've gathered the 5 or so minutes of 2019 data, to get yourself back up and running).

Please note this is probably a driver issue, but it isn't a disk issue - the 333 error is actually caused by the 2019, due to the way the registry is mapped and the fact that it requires NPP (which you've exausted on the server) to complete the registry flush to disk. If you want to know what nonpaged pool is (above and beyond it's a kernel memory pool that drivers use), you can find more in the Windows Internals' books by Mark Russinovich.

Tool download.