0x8E BSOD bugcheck help

[Tripredacus]

My co-worker's computer (around noon time EST) decided to do some BSOD. It is 0x8E pointing to iaStorv.sys (Intel Matrix). His data volume is a 4 drive RAID5. It is healthy in the RAID BIOS. The computer will BSOD after being in Windows after about 2-5 minutes. Logging into the OS is not required. Safe Mode works fine, even when accessing (copy, move, delete files) the RAID volume. I have already updated to the latest driver for the board.

OS: Windows 7 Enterprise SP1 x86

Board: Intel DX58SO

The same BSOD happens with either driver. He has already tried a previous restore point without a problem.

MODULE_NAME: iaStor

FAULTING_MODULE: 82c37000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  4cd505bd

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
iaStor+2ff15
8be3af15 8b4704          mov     eax,dword ptr [edi+4]

TRAP_FRAME:  b000f744 -- (.trap 0xffffffffb000f744)
ErrCode = 00000000
eax=8890e800 ebx=00000000 ecx=0000000e edx=8890e790 esi=8890e790 edi=00000000
eip=8be3af15 esp=b000f7b8 ebp=b000f7c8 iopl=0         nv up ei ng nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010286
iaStor+0x2ff15:
8be3af15 8b4704          mov     eax,dword ptr [edi+4] ds:0023:00000004=????????
Resetting default scope

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x8E

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 82cec01c to 82d15e9c

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
b000f2b4 82cec01c 0000008e c0000005 8be3af15 nt!KeBugCheckEx+0x1e
b000f6d4 82c75e66 b000f6f0 00000000 b000f744 nt!RtlAppendUnicodeToString+0x45d
b000f764 82c05ba9 85e4cc7c 00000000 00000000 nt!Kei386EoiHelper+0x1de
b000f7c8 82c6e5be 864aab50 8890e800 94057888 hal!KfLowerIrql+0x61
b000f7e0 8bf86f2b 8bf8a4a4 82c6e539 00000001 nt!IofCallDriver+0x64
b000f814 8bf92aba b000f8e8 2a861fdf 00000000 MpFilter+0xf2b
b000f8a8 8bf926af c0000004 b000f8e8 00000000 MpFilter+0xcaba
b000f8c4 8bf5519a 85f5fc78 b000f8e8 b000f900 MpFilter+0xc6af
b000f930 8bf5a9ec 85e28008 85f5fc18 3bf5d18d fltmgr!FltDetachVolume+0x832
b000f998 8bf5ac5b 88ceaeb0 88edb4f8 88edb4f8 fltmgr!FltProcessFileLock+0x20a0
b000f9c8 82c6e5be 88ceaeb0 88edb4f8 82d6eb88 fltmgr!FltProcessFileLock+0x230f
b000f9e0 82dd0daf 82c03870 864aab50 82c03900 nt!IofCallDriver+0x64
b000fa44 82cdf57e 864aab50 85f5f301 00000000 nt!NtDeleteFile+0x67c
b000fa7c 82e7cd19 85f5f320 b000fba8 b000fb40 nt!RtlCopyUnicodeString+0x16e
b000fb60 82e5cc2e 864aab50 a5bbf3f8 85f73008 nt!NtClose+0x821
b000fbdc 82e6d040 00000000 b000fc30 00000040 nt!ObCreateObject+0x90b
b000fc38 82e63b1e 00bbe4d0 85bbf3f8 00000001 nt!ObOpenObjectByName+0x165
b000fcb4 82e87396 00bbe52c 80100080 00bbe4d0 nt!NtAllocateVirtualMemory+0x1f52
b000fd00 82c7527a 00bbe52c 80100080 00bbe4d0 nt!NtCreateFile+0x34
b000fd34 76e67094 badb0d00 00bbe498 00000000 nt!ZwYieldExecution+0xb66
b000fd38 badb0d00 00bbe498 00000000 00000000 0x76e67094
b000fd3c 00bbe498 00000000 00000000 00000000 0xbadb0d00
b000fd40 00000000 00000000 00000000 00000000 0xbbe498

STACK_COMMAND:  kb

FOLLOWUP_IP: 
iaStor+2ff15
8be3af15 8b4704          mov     eax,dword ptr [edi+4]

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  iaStor+2ff15

His PC does not have the debugger installed, and it won't let me install it in Safe Mode due to a DCOM error with MSIServer. So I had to analyze on my own PC.

Other than updating the drivers, I can't think what else to try, and I'm not sure what I'm looking for in the dump. Any ideas?

[cluberti]

Mind zipping it up and posting it somewhere? I can provide you FTP space if required.

[Tripredacus]

I've got 2 of them, but we are not ruling out a problem with the HDD. The second dump I gathered was when I ran Gmer on it just got kicks. It reported a failure to read MBR on Disk 0 on the initial scan. I do not think I ever ran Gmer in Safe Mode before, so I do not know if that is why I got that error. I was unable to write down or screenshot what it actually said, because then I got a 0x7E STOP error.

MODULE_NAME: nt

FAULTING_MODULE: 82049000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  4f766ae5

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
nt!wcsupr+2aa
820eb2ba 8b5110          mov     edx,dword ptr [ecx+10h]

EXCEPTION_RECORD:  8c3bbba4 -- (.exr 0xffffffff8c3bbba4)
ExceptionAddress: 820eb2ba (nt!wcsupr+0x000002aa)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 09187910
Attempt to read from address 09187910

CONTEXT:  8c3bb780 -- (.cxr 0xffffffff8c3bb780)
eax=8c2b26f8 ebx=00000004 ecx=09187900 edx=00000258 esi=00000009 edi=8c2b2120
eip=820eb2ba esp=8c3bbc6c ebp=8c3bbc88 iopl=0         nv up ei pl nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010206
nt!wcsupr+0x2aa:
820eb2ba 8b5110          mov     edx,dword ptr [ecx+10h] ds:0023:09187910=????????
Resetting default scope

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x7E

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 820c1ed8 to 820eb2ba

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
8c3bbc88 820c1ed8 00000000 84bb7720 00000000 nt!wcsupr+0x2aa
8c3bbd50 82252056 00000000 ab22cb2c 00000000 nt!ObfDereferenceObjectWithTag+0x221
8c3bbd90 820fa1a9 820c1d60 00000000 00000000 nt!RtlAnsiStringToUnicodeString+0x19d
00000000 00000000 00000000 00000000 00000000 nt!KeInitializeTimerEx+0x3c6


FOLLOWUP_IP: 
nt!wcsupr+2aa
820eb2ba 8b5110          mov     edx,dword ptr [ecx+10h]

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  nt!wcsupr+2aa

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ntkrpamp.exe

STACK_COMMAND:  .cxr 0xffffffff8c3bb780 ; kb

Uploaded dumps.rar to SkyDrive... apparently let me put the whole thing up there.

Edited July 18, 2012 by Tripredacus

[cluberti]

It's hard to say what's happening in the class driver under the Partition Manager, but it looks like while attempting to mount the RAID device the Intel storage driver caused a bugcheck:

0x8E:

// The bugcheck (crashing) stack:
1: kd> kn
  *** Stack trace for last set context - .thread/.cxr resets it
 # ChildEBP RetAddr  
WARNING: Stack unwind information not available. Following frames may be wrong.
00 b000f7c8 82c6e5be iaStor+0x2ff15
01 b000f7e0 8bf86f2b nt!IofCallDriver+0x63
02 b000f814 8bf92aba MpFilter+0xf2b
03 b000f8a8 8bf926af MpFilter+0xcaba
04 b000f8c4 8bf5519a MpFilter+0xc6af
05 b000f930 8bf5a9ec fltmgr!FltpPerformPreMountCallbacks+0x1d0
06 b000f998 8bf5ac5b fltmgr!FltpFsControlMountVolume+0x116
07 b000f9c8 82c6e5be fltmgr!FltpFsControl+0x5b
08 b000f9e0 82dd0daf nt!IofCallDriver+0x63
09 b000fa44 82cdf57e nt!IopMountVolume+0x1d8
0a b000fa7c 82e7cd19 nt!IopCheckVpbMounted+0x64
0b b000fb60 82e5cc2e nt!IopParseDevice+0x7c9
0c b000fbdc 82e6d040 nt!ObpLookupObjectName+0x4fa
0d b000fc38 82e63b1e nt!ObOpenObjectByName+0x165
0e b000fcb4 82e87396 nt!IopCreateFile+0x673
0f b000fd00 82c7527a nt!NtCreateFile+0x34
10 b000fd00 76e67094 nt!KiFastCallEntry+0x12a
11 00bbe534 00000000 0x76e67094

// The IRP for this thread should be in @edx...
1: kd> r
Last set context:
eax=8890e800 ebx=00000000 ecx=0000000e edx=8890e790 esi=8890e790 edi=00000000
eip=8be3af15 esp=b000f7b8 ebp=b000f7c8 iopl=0         nv up ei ng nz na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010286
iaStor+0x2ff15:
8be3af15 8b4704          mov     eax,dword ptr [edi+4] ds:0023:00000004=????????

// ...and shows us in the Intel RST driver:
1: kd> !irp 8890e790 
Irp is active with 1 stacks 1 is current (= 0x8890e800)
 No Mdl: System buffer=85dda0c0: Thread 85e4ca30:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
>[  e, 0]   0  0 864aab50 00000000 00000000-00000000    
	       \Driver\iaStor
			Args: 00000008 00000000 002d0c14 00000000

// The originating IRP is also in iaStor - looks like whatever it was doing, it completed it:
1: kd> !io 8890e790
Irp is active with 1 stacks 1 is current (= 0x8890e800)
 No Mdl: System buffer=85dda0c0: Thread 85e4ca30:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
>[  e, 0]   0  0 864aab50 00000000 00000000-00000000    
	       \Driver\iaStor
			Args: 00000008 00000000 002d0c14 00000000

Notification Event: b000f800 

 [  e, 0] = IRP_MJ_DEVICE_CONTROL, IRP_MN_???

  IO Status: 0 : STATUS_SUCCESS

File Object: 00000000 

Current Driver:
No.  MEMORY_RANGE         CheckSum  TimeStamp                           Flag Author            Image Name       Dist Version                Path
  1  8be0b000 - 8bf0c000  00062058  4cd505bd  Sat Nov 06 00:37:33 2010  ???                   iaStor.sys            \SystemRoot\system32\DRIVERS\iaStor.sys  

// Investigating the device object does point to iaStor, so this is probably accurate:
1: kd> !devobj 864aab50 
Device object (864aab50) is for:
 IAAStorageDevice-1 \Driver\iaStor DriverObject 86565f10
Current Irp 00000000 RefCount 1 Type 00000007 Flags 00005050
Vpb 881de248 Dacl 90200d1c DevExt 00000000 DevObjExt 86569198 Dope 86561d58 DevNode 86597b70 
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0x00000100)  FILE_DEVICE_SECURE_OPEN
AttachedDevice (Upper) 880e43f0 \Driver\Disk
Device queue is not busy.

1: kd> !drvobj 86565f10
Driver object (86565f10) is for:
 \Driver\iaStor
Driver Extension List: (id , addr)

Device Object list:
864aab50  8654a028  86578028  8656c218

// Looking back at the stack for what was being done on this thread, it looks like there was a drive mount happening:
1: kd> .frame b
0b b000fb60 82e5cc2e nt!IopParseDevice+0x7c9

1: kd> dt CompleteName
"\Device\Ide\IAAStorageDevice-1"

// Version of the Intel RST driver:
1: kd> lmvm iastor
start    end        module name
8be0b000 8bf0c000   iaStor     (no symbols)           
    Loaded symbol image file: iaStor.sys
    Image path: \SystemRoot\system32\DRIVERS\iaStor.sys
    Image name: iaStor.sys
    Timestamp:        Sat Nov 06 00:37:33 2010 (4CD505BD)
    CheckSum:         00062058
    ImageSize:        00101000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

// Machine info:
1: kd> !sysinfo machineid
Machine ID Information [From Smbios 2.5, DMIVersion 37, Size=1616]
BiosMajorRelease = 0
BiosMinorRelease = 0
FirmwareMajorRelease = 0
FirmwareMinorRelease = 0
BiosVendor = Intel Corp.
BiosVersion = SOX5810J.86A.5529.2010.1214.2317
BiosReleaseDate = 12/14/2010
SystemManufacturer =                                 
SystemProductName =                                 
SystemVersion =                         
BaseBoardManufacturer = Intel Corporation
BaseBoardProduct = DX58SO
BaseBoardVersion = AAE29331-501

0x7E:

 // Thread running on CPU 0 at the time of this crash:
1: kd> !thread 86fdfd48
THREAD 86fdfd48  Cid 03e8.0dfc  Teb: 7ff89000 Win32Thread: fde56ba8 RUNNING on processor 0
Not impersonating
DeviceMap                 8c008ab8
Owning Process            86b8e508       Image:         svchost.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      106794         Ticks: 0
Context Switch Count      488750         IdealProcessor: 0             
UserTime                  00:00:42.681
KernelTime                00:00:04.695
Win32 Start Address 0x769712e5
Stack Init a1ffdfd0 Current a1ffda48 Base a1ffe000 Limit a1ffb000 Call 0
Priority 8 BasePriority 6 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr  Args to Child              
8078add4 8f810560 862ca028 821b0380 854fa15c USBPORT!USBPORT_ProcessNeoStateChangeList+0x9 (FPO: [Non-Fpo]) (CONV: stdcall)
8078ade8 820ca477 862ca028 00000000 8078ae94 USBPORT!USBPORT_DM_IoTimerDpc+0x20 (FPO: [Non-Fpo]) (CONV: stdcall)
8078ae08 820c3019 821b0360 021b0318 1d9b665b nt!IopTimerDispatch+0x49 (CONV: stdcall)
8078ae4c 820c2fbd 82173d20 8078af78 00000003 nt!KiProcessTimerDpcTable+0x50 (CONV: stdcall)
8078af38 820c2e7a 82173d20 8078af78 00000000 nt!KiProcessExpiredTimerList+0x101 (CONV: stdcall)
8078afac 820c100e 0001a12a a1ffdd34 00000000 nt!KiTimerExpiration+0x25c (CONV: stdcall)
8078aff4 820c07dc a1ffdce4 00000000 00000000 nt!KiRetireDpcList+0xcb (CONV: fastcall)
8078aff8 a1ffdce4 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c (FPO: [Uses EBP] [0,0,1])
WARNING: Frame IP not in any known module. Following frames may be wrong.
820c07dc 00000000 0000001a 00d6850f bb830000 0xa1ffdce4

// The only other active thread at the time, running on CPU 6:
1: kd> !thread 85551d48  
THREAD 85551d48  Cid 0004.011c  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 6
Not impersonating
DeviceMap                 8c008ab8
Owning Process            84afe820       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      105903         Ticks: 891 (0:00:00:13.899)
Context Switch Count      111385         IdealProcessor: 6             
UserTime                  00:00:00.000
KernelTime                00:05:04.311
Win32 Start Address iaStor (0x8b10e424)
Stack Init 8e007fd0 Current 8e007ad0 Base 8e008000 Limit 8e005000 Call 0
Priority 16 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr  Args to Child              
8e007d24 8b10e273 00000000 85551d48 00000000 hal!KfReleaseSpinLock+0x4 (FPO: [0,0,0])
WARNING: Stack unwind information not available. Following frames may be wrong.
8e007d44 8b10e432 8552eb38 8e007d90 82252056 iaStor+0x1c273
8e007d50 82252056 8552eb38 a9190b2c 00000000 iaStor+0x1c432
8e007d90 820fa1a9 8b10e424 8552eb38 00000000 nt!PspSystemThreadStartup+0x9e (CONV: stdcall)
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19

The USBSTOR stack running on CPU 0 is calling _ProcessNeoStateChangeList, which parses USB endpoints to find attached devices and as such requires elevation of IRQL to dispatch mode to parse for those potential endpoints. The iaStor stack of the loading of the iaStor driver (doing something, but not obvious what without symbols) has a dispatch IRQL and is in the process of releasing a SpinLock at the time of the bugcheck. I don't believe that the USB stack is in any way related, because in it's current state, it is waiting for this iaStor thread to finish - in essence, it is doing nothing but waiting.

Ultimately, both of these dumps seem to point to either the Intel driver, or whatever it's mounting. It looks like this is an Intel DX58SO board, and the latest RST driver for that board is the November 2010 release, according to Intel, which it looks like the install is already using. Short of a BIOS update to the May 2012 release, I'm not sure what else you could try short of removing the drives and trying with known good ones. It does look like either a driver or, more likely, drive or array problem.

[Tripredacus]

Thanks for taking a look. It is indeed a DX58SO board, however it is a corporate sample. There are 2 RAID arrays involved besides the RAID5, the other is a RAID1 which holds the boot volume. It is difficult to determine which of these arrays could be causing the problem, other than the MBR read error that Gmer threw. In other testing, the OS will still BSOD when the boot array is degraded (booting with 1 drive). WD Tools flat out refused to test the disks, as it read them as being blank.

I had hoped some concrete evidence would show up in the dumps, but an upgrade may be in order. That may either be a total rebuild of the boot array, and/or upgrading the board. Hopefully it doesn't turn out to be caused by the data volume, but since Safe Mode is fairly stable, backing it up shouldn't be a problem.