System Crashing .... plz help asap

**tmp007** · July 7, 2009

Hello Windows Guru,

I am facing some nasty problem here

I would like to know root cause of the system crash - which system process/object terminates causing windows to crash.

I have opened the dump file (MEMORY_06Jul09_3-54PM.DMP) in windbg.

Below is the result....

======================================

Microsoft ® Windows Debugger Version 6.11.0001.404 X86

Loading Dump File [C:\Documents and Settings\Administrator\Desktop\MEMORY_06Jul09_3-54PM\MEMORY_06Jul09_3-54PM.DMP]

Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

Executable search path is:

Windows XP Kernel Version 2600 (Service Pack 2) MP (2 procs) Free x86 compatible

Product: WinNt, suite: TerminalServer SingleUserTS

Built by: 2600.xpsp_sp2_qfe.070227-2300

Machine Name:

Kernel base = 0x804d7000 PsLoadedModuleList = 0x805624a0

Debug session time: Mon Jul 6 15:52:59.656 2009 (GMT+10)

System Uptime: 0 days 19:41:57.373

Loading Kernel Symbols

...............................................................

................................................................

....................

Loading User Symbols

PEB is paged out (Peb.Ldr = 7ffd800c). Type ".hh dbgerr001" for details

Loading unloaded module list

..................................

*******************************************************************************

* *

* Bugcheck Analysis *

* *

*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck F4, {3, 88575da0, 88575f14, 80604528}

unable to get nt!KiCurrentEtwBufferOffset

unable to get nt!KiCurrentEtwBufferBase

PEB is paged out (Peb.Ldr = 7ffd800c). Type ".hh dbgerr001" for details

Probably caused by : hardware_disk

Followup: MachineOwner

---------

1: kd> !analyze -v

*******************************************************************************

* *

* Bugcheck Analysis *

* *

*******************************************************************************

CRITICAL_OBJECT_TERMINATION (f4)

A process or thread crucial to system operation has unexpectedly exited or been

terminated.

Several processes and threads are necessary for the operation of the

system; when they are terminated (for any reason), the system can no

longer function.

Arguments:

Arg1: 00000003, Process

Arg2: 88575da0, Terminating object

Arg3: 88575f14, Process image file name

Arg4: 80604528, Explanatory message (ascii)

Debugging Details:

------------------

unable to get nt!KiCurrentEtwBufferOffset

unable to get nt!KiCurrentEtwBufferBase

PEB is paged out (Peb.Ldr = 7ffd800c). Type ".hh dbgerr001" for details

PROCESS_OBJECT: 88575da0

IMAGE_NAME: hardware_disk

DEBUG_FLR_IMAGE_TIMESTAMP: 0

FAULTING_MODULE: 00000000

PROCESS_NAME: csrss.exe

EXCEPTION_RECORD: a92be9d8 -- (.exr 0xffffffffa92be9d8)

ExceptionAddress: 75b7b399

ExceptionCode: c0000006 (In-page I/O error)

ExceptionFlags: 00000000

NumberParameters: 3

Parameter[0]: 00000000

Parameter[1]: 75b7b399

Parameter[2]: c000009a

Inpage operation failed at 75b7b399, due to I/O error c000009a

EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".

DEFAULT_BUCKET_ID: DRIVER_FAULT

ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".

EXCEPTION_PARAMETER1: 00000000

EXCEPTION_PARAMETER2: 75b7b399

EXCEPTION_PARAMETER3: c000009a

IO_ERROR: (NTSTATUS) 0xc000009a - Insufficient system resources exist to complete the API.

EXCEPTION_STR: 0xc0000006_c000009a

FAULTING_IP:

+325952f0151dfdc

75b7b399 ?? ???

BUGCHECK_STR: 0xF4_IOERR_C000009A

STACK_TEXT:

a92be520 80634281 000000f4 00000003 88575da0 nt!KeBugCheckEx+0x1b

a92be544 806044e6 80604528 88575da0 88575f14 nt!PspCatchCriticalBreak+0x75

a92be574 804dd99f 88575fe8 c0000006 a92be9b0 nt!NtTerminateProcess+0x7d

a92be574 804e46a7 88575fe8 c0000006 a92be9b0 nt!KiFastCallEntry+0xfc

a92be5f4 80522128 ffffffff c0000006 a92be9f8 nt!ZwTerminateProcess+0x11

a92be9b0 80505460 a92be9d8 00000000 a92bed64 nt!KiDispatchException+0x3a0

a92bed34 804e12a8 0375fbe8 0375fc08 00000000 nt!KiRaiseException+0x175

a92bed50 804dd99f 0375fbe8 0375fc08 00000000 nt!NtRaiseException+0x33

a92bed50 75b7b399 0375fbe8 0375fc08 00000000 nt!KiFastCallEntry+0xfc

WARNING: Frame IP not in any known module. Following frames may be wrong.

0375fff4 00000000 00000000 00000000 00000000 0x75b7b399

STACK_COMMAND: kb

FOLLOWUP_IP:

+325952f0151dfdc

75b7b399 ?? ???

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: hardware_disk

FAILURE_BUCKET_ID: 0xF4_IOERR_C000009A_IMAGE_hardware_disk

BUCKET_ID: 0xF4_IOERR_C000009A_IMAGE_hardware_disk

Followup: MachineOwner

---------

1: kd> !process ffffffff88575da0 3

PROCESS 88575da0 SessionId: 0 Cid: 03bc Peb: 7ffd8000 ParentCid: 038c

DirBase: 20fd0000 ObjectTable: e194ee90 HandleCount: 996.

Image: csrss.exe

VadRoot 87f59568 Vads 165 Clone 0 Private 413. Modified 6987. Locked 0.

DeviceMap e1008620

Token e53ec030

ElapsedTime 19:41:33.734

UserTime 00:00:04.390

KernelTime 00:00:16.640

QuotaPoolUsage[PagedPool] 115384

QuotaPoolUsage[NonPagedPool] 7512

Working Set Sizes (now,min,max) (1161, 50, 345) (4644KB, 200KB, 1380KB)

PeakWorkingSetSize 1343

VirtualSize 67 Mb

PeakVirtualSize 91 Mb

PageFaultCount 28724

MemoryPriority BACKGROUND

BasePriority 13

CommitCharge 508

THREAD 885d0da8 Cid 03bc.03c4 Teb: 7ffde000 Win32Thread: e5be0008 WAIT: (WrLpcReply) UserMode Non-Alertable

885d0f9c Semaphore Limit 0x1

THREAD 885d0b30 Cid 03bc.03c8 Teb: 7ffdd000 Win32Thread: e16dd7b8 WAIT: (UserRequest) UserMode Alertable

885cdde8 SynchronizationEvent

88644320 SynchronizationEvent

885cddb8 SynchronizationEvent

THREAD 885cc020 Cid 03bc.03cc Teb: 7ffdc000 Win32Thread: e5a8deb0 WAIT: (WrLpcReceive) UserMode Non-Alertable

89078c68 Semaphore Limit 0x7fffffff

THREAD 8904bb38 Cid 03bc.03d0 Teb: 7ffdb000 Win32Thread: 00000000 WAIT: (WrLpcReceive) UserMode Non-Alertable

890c0b60 Semaphore Limit 0x7fffffff

THREAD 8854e638 Cid 03bc.03dc Teb: 7ffda000 Win32Thread: e5bd69f8 WAIT: (WrLpcReceive) UserMode Non-Alertable

89078c68 Semaphore Limit 0x7fffffff

THREAD 8856fa20 Cid 03bc.03e0 Teb: 7ffd9000 Win32Thread: e123e598 WAIT: (WrUserRequest) KernelMode Alertable

88ffa418 SynchronizationEvent

885cd658 SynchronizationEvent

8905ee08 NotificationTimer

886270a0 SynchronizationEvent

80568420 NotificationEvent

885ce280 SynchronizationEvent

886445e8 SynchronizationTimer

THREAD 88571688 Cid 03bc.03e4 Teb: 7ffd7000 Win32Thread: e4bf4008 WAIT: (WrUserRequest) UserMode Non-Alertable

8856f3e0 SynchronizationEvent

8904f978 SynchronizationEvent

88ff69b0 SynchronizationEvent

THREAD 88529020 Cid 03bc.0414 Teb: 7ffd6000 Win32Thread: e14cca50 WAIT: (WrUserRequest) UserMode Non-Alertable

885543c8 SynchronizationEvent

88ff3b30 SynchronizationEvent

THREAD 88509020 Cid 03bc.057c Teb: 7ffd5000 Win32Thread: e1b0c0c8 WAIT: (WrLpcReceive) UserMode Non-Alertable

89078c68 Semaphore Limit 0x7fffffff

THREAD 8833b508 Cid 03bc.0154 Teb: 7ffd4000 Win32Thread: e175cc30 RUNNING on processor 1

THREAD 8833d788 Cid 03bc.01a4 Teb: 7ffaf000 Win32Thread: e174bc90 WAIT: (WrUserRequest) UserMode Non-Alertable

88342de0 SynchronizationEvent

THREAD 89d4a650 Cid 03bc.0318 Teb: 7ffae000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable

89d64f54 NotificationEvent

--------------

1: kd> .exr 0xffffffffa92be9d8

ExceptionAddress: 75b7b399

ExceptionCode: c0000006 (In-page I/O error)

ExceptionFlags: 00000000

NumberParameters: 3

Parameter[0]: 00000000

Parameter[1]: 75b7b399

Parameter[2]: c000009a

Inpage operation failed at 75b7b399, due to I/O error c000009a

=======================================================================

Please help me to find root cause of crash on ThinkPad Laptop.

Thanks for your help in advance.

**nitroshift** · July 7, 2009

The answer lies here:

EXCEPTION_RECORD: a92be9d8 -- (.exr 0xffffffffa92be9d8)
ExceptionAddress: 75b7b399
ExceptionCode: c0000006 (In-page I/O error)
ExceptionFlags: 00000000
NumberParameters: 3
Parameter[0]: 00000000
Parameter[1]: 75b7b399
Parameter[2]: c000009a
Inpage operation failed at 75b7b399, due to I/O error c000009a
EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
DEFAULT_BUCKET_ID: DRIVER_FAULT
ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".

meaning that the hard-disk is approaching its death. At this point I would take it out, slave to a pc and save any data you want to preserve. Good luck!

**tmp007** · July 7, 2009

Can we find out which process or application cause this crashed?

It appears that csrss.exe was terminated... but why?

Can we exactly pin-point the cause ?

**Kelsenellenelvian** · July 7, 2009

He told you the cause, your hard drive is almost toast.

You will get random failures from now on, untill it dies.

**Mr Snrub** · July 7, 2009

Actually, it might be some system resource getting exhausted... as you found, csrss.exe was the critical process that got killed:

CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been terminated.
Several processes and threads are necessary for the operation of the system; when they are terminated (for any reason), the system can no longer function.
Arguments:
Arg1: 00000003, Process
Arg2: 88575da0, Terminating object
Arg3: 88575f14, Process image file name
Arg4: 80604528, Explanatory message (ascii)
PROCESS_OBJECT: 88575da0
1: kd> !process ffffffff88575da0 3
PROCESS 88575da0 SessionId: 0 Cid: 03bc Peb: 7ffd8000 ParentCid: 038c
DirBase: 20fd0000 ObjectTable: e194ee90 HandleCount: 996.
Image: csrss.exe

The line I think of interest, and its breakdown:

Inpage operation failed at 75b7b399, due to I/O error c000009a
EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
IO_ERROR: (NTSTATUS) 0xc000009a - Insufficient system resources exist to complete the API.

And the "failed at" address is the module address in the thread that raised the exception (the process, csrss.exe):

STACK_TEXT:
a92be520 80634281 000000f4 00000003 88575da0 nt!KeBugCheckEx+0x1b
a92be544 806044e6 80604528 88575da0 88575f14 nt!PspCatchCriticalBreak+0x75
a92be574 804dd99f 88575fe8 c0000006 a92be9b0 nt!NtTerminateProcess+0x7d
a92be574 804e46a7 88575fe8 c0000006 a92be9b0 nt!KiFastCallEntry+0xfc
a92be5f4 80522128 ffffffff c0000006 a92be9f8 nt!ZwTerminateProcess+0x11
a92be9b0 80505460 a92be9d8 00000000 a92bed64 nt!KiDispatchException+0x3a0
a92bed34 804e12a8 0375fbe8 0375fc08 00000000 nt!KiRaiseException+0x175
a92bed50 804dd99f 0375fbe8 0375fc08 00000000 nt!NtRaiseException+0x33
a92bed50 75b7b399 0375fbe8 0375fc08 00000000 nt!KiFastCallEntry+0xfc
WARNING: Frame IP not in any known module. Following frames may be wrong.
0375fff4 00000000 00000000 00000000 00000000 0x75b7b399

I would guess the page in the virtual address space for csrss.exe was paged out to disk, then at some point a context switch occurred to continue executing which incurred the inpage operation - but when pulling the data from disk the I/O failed, making the thread go boom, which terminates the process, and it was a critical process so we bugcheck.

Most commonly in my experience the cause of failing inpage operations is a disk or disk controller failure (the device suddenly vanishes from the system), sometimes due to a driver fault or an I/O mode setting in the BIOS (e.g. AHCI being used)... however here there is the extra bit of info "Insufficient system resources exist to complete the API".

The output from !vm might be useful, to see if it's pool memory or PTE shortage - of course there's a chance it could be a bogus status code if the origin is a dodgy CPU or heat related...

Not running SP3?

**tmp007** · July 7, 2009

Thanks Mr Snrub !

Here is the output you requested.

1: kd> !vm

*** Virtual Memory Usage ***

Physical Memory: 521819 ( 2087276 Kb)

Page File: \??\C:\pagefile.sys

Current: 2095104 Kb Free Space: 2055696 Kb

Minimum: 2095104 Kb Maximum: 4190208 Kb

Available Pages: 213762 ( 855048 Kb)

ResAvail Pages: 423764 ( 1695056 Kb)

Locked IO Pages: 74 ( 296 Kb)

Free System PTEs: 119540 ( 478160 Kb)

Free NP PTEs: 0 ( 0 Kb)

Free Special NP: 0 ( 0 Kb)

Modified Pages: 484 ( 1936 Kb)

Modified PF Pages: 484 ( 1936 Kb)

NonPagedPool Usage: 65534 ( 262136 Kb)

NonPagedPool Max: 65536 ( 262144 Kb)

********** Excessive NonPaged Pool Usage *****

PagedPool 0 Usage: 28165 ( 112660 Kb)

PagedPool 1 Usage: 1713 ( 6852 Kb)

PagedPool 2 Usage: 1690 ( 6760 Kb)

PagedPool 3 Usage: 1682 ( 6728 Kb)

PagedPool 4 Usage: 1670 ( 6680 Kb)

PagedPool Usage: 34920 ( 139680 Kb)

PagedPool Maximum: 91136 ( 364544 Kb)

********** 19498 pool allocations have failed **********

Session Commit: 401 ( 1604 Kb)

Shared Commit: 5977 ( 23908 Kb)

Special Pool: 0 ( 0 Kb)

Shared Process: 6784 ( 27136 Kb)

PagedPool Commit: 34920 ( 139680 Kb)

Driver Commit: 3706 ( 14824 Kb)

Committed pages: 254596 ( 1018384 Kb)

Commit limit: 1006752 ( 4027008 Kb)

Total Private: 155603 ( 622412 Kb)

1474 firefox.exe 45953 ( 183812 Kb)

0a94 Rtvscan.exe 13436 ( 53744 Kb)

1134 wlmail.exe 9173 ( 36692 Kb)

0fcc explorer.exe 7287 ( 29148 Kb)

0bf8 java.exe 7269 ( 29076 Kb)

1e08 java.exe 6639 ( 26556 Kb)

0530 svchost.exe 4617 ( 18468 Kb)

1a68 issimgui.exe 4323 ( 17292 Kb)

1518 PCSuite.exe 3282 ( 13128 Kb)

03d4 winlogon.exe 2930 ( 11720 Kb)

1954 NclBCBTSrv.exe 2606 ( 10424 Kb)

03a8 issimsvc.exe 2079 ( 8316 Kb)

066c svchost.exe 1975 ( 7900 Kb)

0260 c4ebreg.exe 1958 ( 7832 Kb)

0844 hpqtra08.exe 1886 ( 7544 Kb)

00f8 svchost.exe 1711 ( 6844 Kb)

06a0 spoolsv.exe 1641 ( 6564 Kb)

0c84 BTSTAC~1.EXE 1635 ( 6540 Kb)

05fc SPBBCSvc.exe 1582 ( 6328 Kb)

0c34 YahooAUService. 1396 ( 5584 Kb)

0180 ccEvtMgr.exe 1392 ( 5568 Kb)

0320 ccProxy.exe 1270 ( 5080 Kb)

07c0 ccSetMgr.exe 1133 ( 4532 Kb)

0c7c AcSvc.exe 1083 ( 4332 Kb)

040c lsass.exe 1083 ( 4332 Kb)

03a4 ISSVC.exe 1059 ( 4236 Kb)

0850 acs.exe 1056 ( 4224 Kb)

00e8 svchost.exe 1018 ( 4072 Kb)

03ec wweb32.exe 999 ( 3996 Kb)

0750 cisvc.exe 981 ( 3924 Kb)

15bc artcore.exe 952 ( 3808 Kb)

0834 BTTray.exe 919 ( 3676 Kb)

0f78 VPTray.exe 901 ( 3604 Kb)

0b0c SymSPort.exe 899 ( 3596 Kb)

04d4 svchost.exe 829 ( 3316 Kb)

1e20 UEDIT32.EXE 768 ( 3072 Kb)

0508 svchost.exe 743 ( 2972 Kb)

09dc svchost.exe 714 ( 2856 Kb)

0d08 PCS_AGNT.EXE 709 ( 2836 Kb)

02fc isamtray.exe 707 ( 2828 Kb)

0a68 hpqste08.exe 660 ( 2640 Kb)

0570 svchost.exe 627 ( 2508 Kb)

083c svchost.exe 611 ( 2444 Kb)

0770 jqs.exe 589 ( 2356 Kb)

04e4 SNDSrvc.exe 580 ( 2320 Kb)

0734 svchost.exe 573 ( 2292 Kb)

0554 btwdins.exe 558 ( 2232 Kb)

0400 services.exe 537 ( 2148 Kb)

0728 AppleMobileDevi 514 ( 2056 Kb)

0798 DefWatch.exe 513 ( 2052 Kb)

03bc csrss.exe 508 ( 2032 Kb)

0978 SavRoam.exe 478 ( 1912 Kb)

1908 NclUSBSrv.exe 468 ( 1872 Kb)

0dfc SvcGuiHlpr.exe 444 ( 1776 Kb)

0114 svchost.exe 436 ( 1744 Kb)

0a0c cidaemon.exe 423 ( 1692 Kb)

0630 svchost.exe 415 ( 1660 Kb)

11b8 cmd.exe 406 ( 1624 Kb)

013c cmd.exe 406 ( 1624 Kb)

04fc AcPrfMgrSvc.exe 404 ( 1616 Kb)

0884 NetCfgSv.EXE 403 ( 1612 Kb)

1560 ServiceLayer.ex 354 ( 1416 Kb)

0774 CDSWinSrv.exe 330 ( 1320 Kb)

0950 svchost.exe 329 ( 1316 Kb)

05b8 alg.exe 325 ( 1300 Kb)

0bac wrtService.exe 214 ( 856 Kb)

1504 NclRSSrv.exe 188 ( 752 Kb)

1708 NclIrSrv.exe 169 ( 676 Kb)

0b50 TPHDEXLG.exe 162 ( 648 Kb)

04b4 ibmpmsvc.exe 150 ( 600 Kb)

1244 artifdown.exe 112 ( 448 Kb)

080c ntmulti.exe 74 ( 296 Kb)

038c smss.exe 42 ( 168 Kb)

0004 System 8 ( 32 Kb)

1e0c W32MAIN2.EXE 0 ( 0 Kb)

1dc0 W32MAIN2.EXE 0 ( 0 Kb)

1d90 W32MAIN2.EXE 0 ( 0 Kb)

1d30 W32MAIN2.EXE 0 ( 0 Kb)

1cc0 W32MAIN2.EXE 0 ( 0 Kb)

1aac W32MAIN2.EXE 0 ( 0 Kb)

1a98 W32MAIN2.EXE 0 ( 0 Kb)

1494 W32MAIN2.EXE 0 ( 0 Kb)

13b8 cmd.exe 0 ( 0 Kb)

1200 WINWORD.EXE 0 ( 0 Kb)

1168 W32MAIN2.EXE 0 ( 0 Kb)

0e40 pcssnd.exe 0 ( 0 Kb)

0618 cmd.exe 0 ( 0 Kb)

0528 W32MAIN2.EXE 0 ( 0 Kb)

Secondly. its not SP3. Its Windows XP Kernel Version 2600 (Service Pack 2) MP (2 procs) Free x86 compatible

**Mr Snrub** · July 7, 2009

NonPagedPool Usage: 65534 ( 262136 Kb)
NonPagedPool Max: 65536 ( 262144 Kb)
********** Excessive NonPaged Pool Usage *****
********** 19498 pool allocations have failed **********

Nonpaged pool totally exhausted, something has leaked.

The output from !poolused 7 will be long - it is sorted in descending order in nonpaged bytes, so the first few lines are the most interesting.

This will give a clue as to the pooltags used for the allocations, and maybe a direct indicator as to who might have made them.

AV filter drivers are common leakers of pool memory - what AV do you have installed?

My comment on SP3 was intended as: "why isn't SP3 installed?"

**cluberti** · July 7, 2009

NonPagedPool Usage: 65534 ( 262136 Kb)
NonPagedPool Max: 65536 ( 262144 Kb)
********** Excessive NonPaged Pool Usage *****

Well, that is pretty bad. I'm not surprised your box crashed, considering how fragmented your NPP likely is (on top of being almost completely used). Can you put the .dmp file up (zipped or RAR'ed) somewhere we can get to it?

Edited July 7, 2009 by cluberti
Mr Snrub beat me to it by 1 minute!!!! lol

**tmp007** · July 7, 2009

Thanks guys for considering this high priority.

1) Here is what I found : http://blogs.msdn.com/oldnewthing/archive/...04/9172708.aspx

2) Can you explain in details what the below means? curious to know what those number indicates too.

========

NonPagedPool Usage: 65534 ( 262136 Kb)

NonPagedPool Max: 65536 ( 262144 Kb)

********** Excessive NonPaged Pool Usage *****

===========

3) 1: kd> !poolused 7

unable to get PoolTrackTable - pool tagging is disabled, enable it to use this command

Use gflags.exe and check the box that says "Enable pool tagging".

4) Due to size limitation I have uploaded the "Mini070609-01.dmp"

Plz help me to pin pt the exact cause and way to fix it !

Mini070609_01.rar

**cluberti** · July 7, 2009

I'm not sure if this dump is going to give us much info, as it's a minidump. It's only going to have information (basic) about the currently executing thread (in csrss.exe) and the registers, the memory information is not here.

Would it be possible to get a full (complete) memory dump?

**tmp007** · July 7, 2009

Full Dump available here > http://www.megaupload.com/?d=VBI4U0LR

**cluberti** · July 8, 2009

Darn - you did say this was XP SP2, didn't you. Pool tagging isn't enabled by default (as the error said) until Server 2003 or higher, so you'll need to enable pool tagging and get another dump. To enable pool tagging, first you need to download and install the current release of the debugging tools for windows, then go to the directory (usually C:\Program Files\Debugging Tools for Windows (might have an x86 at the end of the folder name) and run gflags.exe. Once it's open, check this box, then reboot:

Once you reboot, get another dump - we'll be able to tell you what's consuming Nonpaged Pool from a dump with this flag enabled.

**tmp007** · July 8, 2009

My observation is that it crashes after 10 hours once I keep the machine up and running.

I have enabled the pool tagging and I will collect the dump and will let you know.

Thanks for your continual efforts.

**Mr Snrub** · July 8, 2009

While we wait for the dump with pool tagging enabled...

2) Can you explain in details what the below means? curious to know what those number indicates too.
========
NonPagedPool Usage: 65534 ( 262136 Kb)
NonPagedPool Max: 65536 ( 262144 Kb)
********** Excessive NonPaged Pool Usage *****
===========

Nonpaged (or nonpageable) pool memory is for dynamic memory allocations in the kernel that cannot be paged out to disk - drivers have to use this pool for data that must be available at all times, as an page fault (request for a virtual page not resident in physical RAM, but in the page file on disk) is not allowed when they have control.

This is the classic IRQL_NOT_LESS_THAN_OR_EQUAL bugcheck, if the driver developer makes this assumption.

Because the nonpaged pool region has to take physical memory, and is a subset of the 2GB kernel space, its absolute maximum is capped at 256MB (but systems with less than ~768MB RAM, or using /3GB would have less than this as their limit).

Because it is a finite system resource, once it is no longer required an allocation is meant to be returned to the pool by marking is as free.

(The other, larger pool is paged pool - this is the same concept of dynamic memory allocations in the kernel, but these ones are non-critical data that we can put into the page file as needed to free physical memory.)

What do you have in the way of USB devices connected to the machine?

I ask because I had a poke around the nonpaged pool region to see if there are any clues, and saw a lot of Irps (I/O request packets), and so ran the !irpfind command to get a summary:

1: kd> !irpfind
unable to get large pool allocation table - either wrong symbols or pool tagging is disabled
Searching NonPaged pool (827b6000 : 8a7b6000) for Tag: Irp?
  Irp	[ Thread ] irpStack: (Mj,Mn)   DevObj  [Driver]		 MDL Process
827b64a8 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827b6b28 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827b8008 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827b83c0 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827b8b20 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827b9008 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827b9d98 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827bad98 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
827bb008 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
...
ffbddb28 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
ffbde008 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
ffbde3d8 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
ffbde648 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
ffbdeb28 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)
ffbded98 [00000000] Irp is complete (CurrentLocation 4 > StackCount 3)

There are 148,962 Irps listed in the output in total.

Taking a look at the first in the list... !pool lets us confirm the allocation is from nonpaged pool and is an IRP, then !irp can give us some details on the I/O taking place, and !devstack lets us see the underlying device:

1: kd> !pool 827b64a8 
Pool page 827b64a8 region is Nonpaged pool
 827b6000 size:  270 previous size:	0  (Allocated)   P_. (Protected)
 827b6270 size:  230 previous size:  270  (Free)	   ....
*827b64a0 size:  270 previous size:  230  (Allocated) *Irp 
		Pooltag Irp  : Io, IRP packets
 827b6710 size:  270 previous size:  270  (Allocated)  ..3. (Protected)
 827b6980 size:  1a0 previous size:  270  (Free)	   Attv
 827b6b20 size:  270 previous size:  1a0  (Allocated)  Irp 
 827b6d90 size:  270 previous size:  270  (Allocated)   P_. (Protected)

1: kd> !irp 827b64a8 
Irp is active with 3 stacks 4 is current (= 0x827b6584)
 No Mdl: No System Buffer: Thread 00000000:  Irp is completed.  
	 cmd  flg cl Device   File	 Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000	
			Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000	
			Args: 00000000 00000000 00000000 00000000
 [  f, 0]   0  0 89764618 00000000 bad750ac-89763748	
		   \Driver\usbuhci	usbhub!USBH_FdoIdleNotificationRequestComplete
			Args: 00000000 00000000 00000000 00000000

1: kd> !devstack 89764618 
  !DevObj   !DrvObj			!DevExt   ObjectName
  89763690  \Driver\usbhub	 89763748  000000f6
> 89764618  \Driver\usbuhci	897646d0  USBPDO-0
!DevNode 89b2fa90 :
  DeviceInst is "USB\ROOT_HUB\4&56cb44e&0"
  ServiceName is "usbhub"

I can see some processes that hint at something related to communications (USB, IrDA, Bluetooth):

PROCESS 884f5020  SessionId: 0  Cid: 0554	Peb: 7ffd9000  ParentCid: 0400
	DirBase: 2f333000  ObjectTable: e15bd2e8  HandleCount:  62.
	Image: btwdins.exe

PROCESS 88043430  SessionId: 0  Cid: 0c84	Peb: 7ffdf000  ParentCid: 04d4
	DirBase: 3dc31000  ObjectTable: e7f42c78  HandleCount: 235.
	Image: BTSTAC~1.EXE

PROCESS facf5020  SessionId: 0  Cid: 1908	Peb: 7ffde000  ParentCid: 1560
	DirBase: 5d729000  ObjectTable: e8e88850  HandleCount:  67.
	Image: NclUSBSrv.exe

PROCESS fa91c8c0  SessionId: 0  Cid: 1954	Peb: 7ffd9000  ParentCid: 1560
	DirBase: 47e38000  ObjectTable: e16a4260  HandleCount: 145.
	Image: NclBCBTSrv.exe

PROCESS f9f7c020  SessionId: 0  Cid: 1708	Peb: 7ffd8000  ParentCid: 1560
	DirBase: 7ed19000  ObjectTable: e17ba830  HandleCount:  47.
	Image: NclIrSrv.exe

PROCESS facf0020  SessionId: 0  Cid: 1504	Peb: 7ffdf000  ParentCid: 1560
	DirBase: 46b65000  ObjectTable: e67a6b60  HandleCount:  45.
	Image: NclRSSrv.exe

And then there's always AV to consider:

a6c30000 a6c441e0   naveng   \??\C:\PROGRA~1\COMMON~1\SYMANT~1\VIRUSD~1\20090705.003\naveng.sys
a6c45000 a6d19440   navex15  \??\C:\PROGRA~1\COMMON~1\SYMANT~1\VIRUSD~1\20090705.003\navex15.sys
a9c23000 a9c40000   EraserUtilRebootDrv \??\C:\Program Files\Common Files\Symantec Shared\EENGINE\EraserUtilRebootDrv.sys
a9c40000 a9c9e000   eeCtrl   \??\C:\Program Files\Common Files\Symantec Shared\EENGINE\eeCtrl.sys
a9d60000 a9dc2000   SPBBCDrv \??\C:\Program Files\Common Files\Symantec Shared\SPBBC\SPBBCDrv.sys
a9e2c000 a9e6e000   symidsco \??\C:\PROGRA~1\COMMON~1\SYMANT~1\SymcData\SCFIDS~1\20090625.001\symidsco.sys
a9e6e000 a9e97000   SYMFW	\SystemRoot\System32\Drivers\SYMFW.SYS
aa19a000 aa1ae000   Savrtpel \??\C:\Program Files\Symantec Client Security\Symantec AntiVirus\Savrtpel.sys
aa1ae000 aa1d0000   SYMEVENT \??\C:\Program Files\Symantec\SYMEVENT.SYS
aa1d0000 aa228000   savrt	\??\C:\Program Files\Symantec Client Security\Symantec AntiVirus\savrt.sys

First rule of troubleshooting a new problem - did you change or install anything recently?

In particular anything related to USB, bluetooth or chipset drivers?

Maybe mobile phone sync software, or even fingerprint scanner drivers?

Secondly, try to reduce the problem to its bare minimum - is there a particular piece of software that causes the problem to occur?

Whilst running without AV is not a long-term solution, it's a valid test for problems that occur routinely - I would uninstall the Symantec software and see if the symptom disappears (note: disabling is not the same as uninstalling, the kernel drivers are still present and get involved in I/O).

**malmal** · July 10, 2009

Sooo... What happened? Was it resolved?

Sign In

System Crashing .... plz help asap

Recommended Posts

tmp007

nitroshift

tmp007

Kelsenellenelvian

Mr Snrub

tmp007

Mr Snrub

cluberti

tmp007

cluberti

tmp007

cluberti

tmp007

Mr Snrub

malmal

Please sign in to comment

Recently Browsing 0 members

Activity

Browse