Bad boy Warrior Posted August 5, 2008 Posted August 5, 2008 Just wondering if anyone has any ideas on this problem. A SQL job runs and it freezes up the whole system. We have to switch it off manually and looking at the event viewer we see 0x8000005 and odd 0xa000000 messages listed.It an HP system with 32 bit op system but running on 64 bit harware (8GB ram).Any ideas on resolving this? or pointing towards the right direction?Thanks
CoffeeFiend Posted August 5, 2008 Posted August 5, 2008 We have to switch it off manually and looking at the event viewer we see 0x8000005 and odd 0xa000000 messages listed.You're seeing these because the machine wasn't shutdown properly. It doesn't give any useful infos.Just wondering if anyone has any ideas on this problem. A SQL job runs and it freezes up the whole system.Hard to say what happens with so little information. Does it freeze hard? (as in the whole server) -- if so, enable memory dumps so we can try to analyze them. Or is it just MS SQL that stops processing queries? (could be high load simply, you looked at CPU load/performance counters?)What's the actual query and table schema? How many rows in the table? (very useful to know, so we can guess just how resource intensive to process, and even try to duplicate it) What version of MS SQL? Is the box fully patched & using the latest drivers?We just can't solve the problem or guess what is it based on so little information unfortunately.
Bad boy Warrior Posted August 5, 2008 Author Posted August 5, 2008 We have to switch it off manually and looking at the event viewer we see 0x8000005 and odd 0xa000000 messages listed.You're seeing these because the machine wasn't shutdown properly. It doesn't give any useful infos.Just wondering if anyone has any ideas on this problem. A SQL job runs and it freezes up the whole system.Hard to say what happens with so little information. Does it freeze hard? (as in the whole server) -- if so, enable memory dumps so we can try to analyze them. Or is it just MS SQL that stops processing queries? (could be high load simply, you looked at CPU load/performance counters?)What's the actual query and table schema? How many rows in the table? (very useful to know, so we can guess just how resource intensive to process, and even try to duplicate it) What version of MS SQL? Is the box fully patched & using the latest drivers?We just can't solve the problem or guess what is it based on so little information unfortunately.Yes it freezes hard but was about to use Cluberti's method to get a memory dump(Memory dump from an application/process that is HANGING (not crashing):1. Create a directory called c:\adplus2. Open a command prompt and change to the directory where you installed the debugging tools. By default, this is "C:\Program Files\Debugging Tools for Windows"3. Type the following command in the command prompt:cscript adplus.vbs -hang -pn appname -quiet -o c:\adplus (where "appname" is the .exe name of the application that is hanging)4. Once the debugger has finished (this can take some time), the command prompt window(s) will close, and you will have data in your C:\adplus folder that can be analyzed.)It may be a SQL issue as looking at the CPU usage it consumes almost half the memory and then suddenly the CPU has no activity....Box is almost fully patched i.e. it doesnt have the latest SQL 2005 SP2 update (it does has SP2 for SQL 2005 though). Ill get the table info and post if the memory dumps arent good enough.Thanks
CoffeeFiend Posted August 5, 2008 Posted August 5, 2008 (edited) Yes it freezes hard but was about to use Cluberti's method to get a memory dumpThen you're saying it's just SQL Server freezing, not Windows (which is what I meant). If windows was frozen, you wouldn't be able to use his method to get dumps...If it's just SQL Server that stops responding, most likely it's just a query you're running that's too intensive, and it's just very busy trying to execute it. Some bad queries will tend to "freeze" a lot of different RDBMS'es (make them unresponsive for a while). Check your performance counters, and see what's happening (CPU pegged? I/O bottleneck? paging memory a lot? etc) I'm not sure who's writing the queries being ran on it, but some beginners will write pretty bad queries that will do this kind of stuff: table scans on large tables, lots of joins, selecting unnecessary columns, no/bad indexes, poor choices in terms of database schema and data types, etc resulting in poor execution plans, high resource usage and all that. This is why I was asking about that other stuff. Edited August 5, 2008 by crahak
Bad boy Warrior Posted August 5, 2008 Author Posted August 5, 2008 Yes it freezes hard but was about to use Cluberti's method to get a memory dumpThen you're saying it's just SQL Server freezing, not Windows (which is what I meant). If windows was frozen, you wouldn't be able to use his method to get dumps...Its a bit strange....the process sqlserver.exe has high memory consumption and eventually the CPU shows no activity. But i can minimise windows etc and eventually i cant do anything for around 10 mins so switch off the server. Now for some reason it ran fine without any problems but when we ran it again the same issue occured.It could be the query but how could i definately say that it is the problem as it was created by another outside source??Thanks
CoffeeFiend Posted August 5, 2008 Posted August 5, 2008 the process sqlserver.exe has high memory consumptionThat's normal. It tries to keep data in memory, because it's a LOT faster to return results from data in memory than having to read it from disk first.and eventually the CPU shows no activityWell, maybe the query finished running, or maybe it did hang indeed.Now for some reason it ran fine without any problems but when we ran it again the same issue occured.Ok, now my #1 priority question would be, what changed since then? Any new apps, or new versions of some app or such you're using (that could make bad queries), or changes you've made recently to anything? If it's only been doing that for a couple days, and that something was changed a couple days ago, it might be a good idea to investigate.Something else that is very much worth looking into is SQL Server query logging (there should be plenty of details about that in BOL, technet or MSDN). That will log the name of the database the query is ran against, the user who ran it, the query itself, its execution time and some more helpful data. It would help figuring out what's going on for sure.
Bad boy Warrior Posted August 5, 2008 Author Posted August 5, 2008 (edited) Thanks ill look into that but is the SQL profiler the same? Nothings really changed except the query has had indexes added to speed up the processalso one of the logs that i have from running the dump tools (whether thsi makes a difference or not) follows: 0 32 0 System Process Command Line: 0 32 4 System Command Line: 0 32 344 smss.exe Command Line: \SystemRoot\System32\smss.exe 0 32 392 csrss.exe Command Line: C:\WINDOWS\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16 0 32 420 winlogon.exe Command Line: winlogon.exe 0 32 468 services.exe Svcs: Eventlog,PlugPlay Command Line: C:\WINDOWS\system32\services.exe 0 32 480 lsass.exe Svcs: HTTPFilter,Netlogon,PolicyAgent,ProtectedStorage,SamSs Command Line: C:\WINDOWS\system32\lsass.exe 0 32 640 svchost.exe Svcs: DcomLaunch Command Line: C:\WINDOWS\system32\svchost.exe -k DcomLaunch 0 32 708 svchost.exe Svcs: RpcSs Command Line: C:\WINDOWS\system32\svchost.exe -k rpcss 0 32 800 Smc.exe Svcs: SmcService Command Line: "C:\Program Files\Symantec\Symantec Endpoint Protection\Smc.exe" 0 32 816 svchost.exe Svcs: Dhcp,Dnscache Command Line: C:\WINDOWS\system32\svchost.exe -k NetworkService 0 32 868 svchost.exe Svcs: Alerter,LmHosts,W32Time Command Line: C:\WINDOWS\system32\svchost.exe -k LocalService 0 32 884 svchost.exe Svcs: AeLookupSvc,BITS,Browser,CryptSvc,dmserver,EventSystem,helpsvc,lanmanserver,lanmanworkstation,Netman,Nla,RasMan,Schedule,seclogon,SENS,ShellHWDetection,TrkWks,winmgmt,wuauserv,WZCSVC Command Line: C:\WINDOWS\System32\svchost.exe -k netsvcs 0 32 960 ccSvcHst.exe Svcs: ccEvtMgr,ccSetMgr Command Line: "C:\Program Files\Common Files\Symantec Shared\ccSvcHst.exe" /h ccCommon 0 32 1436 spoolsv.exe Svcs: Spooler Command Line: C:\WINDOWS\system32\spoolsv.exe 0 32 1464 msdtc.exe Svcs: MSDTC Command Line: C:\WINDOWS\system32\msdtc.exe 0 32 1576 cissesrv.exe Svcs: Cissesrv Command Line: "C:\Program Files\HP\Cissesrv\cissesrv.exe" 0 32 1588 cpqrcmc.exe Svcs: CpqRcmc Command Line: C:\WINDOWS\system32\cpqrcmc.exe 0 32 1604 vcagent.exe Svcs: cpqvcagent Command Line: C:\hp\hpsmh\data\cgi-bin\vcagent\vcagent.exe 0 32 1648 svchost.exe Svcs: ERSvc Command Line: C:\WINDOWS\System32\svchost.exe -k WinErr 0 32 1788 inetinfo.exe Svcs: IISADMIN,MSFtpsvc Command Line: C:\WINDOWS\system32\inetsrv\inetinfo.exe 0 32 1832 MsDtsSrvr.exe Svcs: MsDtsServer Command Line: "C:\Program Files\Microsoft SQL Server\90\DTS\Binn\MsDtsSrvr.exe" 0 32 1992 msftesql.exe Svcs: msftesql$SERVERNAME Command Line: "C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\msftesql.exe" -s:MSSQL.1 -f:SERVERNAME 0 32 2024 msmdsrv.exe Svcs: MSOLAP$SERVERNAME Command Line: "C:\Program Files\Microsoft SQL Server\MSSQL.2\OLAP\bin\msmdsrv.exe" -s "C:\Program Files\Microsoft SQL Server\MSSQL.2\OLAP\Config" 0 32 2164 sqlservr.exe Svcs: MSSQL$BKUPEXEC Command Line: "C:\Program Files\Microsoft SQL Server\MSSQL.4\MSSQL\Binn\sqlservr.exe" -sBKUPEXEC 0 32 2244 sqlservr.exe Svcs: MSSQL$SERVERNAME Command Line: "C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe" -sSERVERNAME 0 32 2288 svchost.exe Svcs: RemoteRegistry Command Line: C:\WINDOWS\system32\svchost.exe -k regsvc 0 32 2304 ReportingServicesService.exe Svcs: ReportServer$SERVERNAME Command Line: "C:\Program Files\Microsoft SQL Server\MSSQL.3\Reporting Services\ReportServer\bin\ReportingServicesService.exe" 0 32 2520 snmp.exe Svcs: SNMP Command Line: C:\WINDOWS\System32\snmp.exe 0 32 2536 sqlbrowser.exe Svcs: SQLBrowser Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlbrowser.exe" 0 32 2588 sqlwriter.exe Svcs: SQLWriter Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe" 0 32 2640 Rtvscan.exe Svcs: Symantec AntiVirus Command Line: "C:\Program Files\Symantec\Symantec Endpoint Protection\Rtvscan.exe" 0 32 2692 sysdown.exe Svcs: sysdown Command Line: C:\WINDOWS\system32\sysdown.exe 0 32 2752 smhstart.exe Svcs: SysMgmtHp Command Line: C:\hp\hpsmh\bin\smhstart.exe 0 32 2952 hpsmhd.exe Command Line: C:\hp\hpsmh\bin\hpsmhd.exe -fC:/hp/hpsmh/conf/smhpd.conf 0 32 2972 beremote.exe Svcs: BackupExecAgentAccelerator Command Line: "C:\Program Files\Symantec\Backup Exec\beremote.exe" 0 32 3216 rotatelogs.exe Command Line: C:\hp\hpsmh\bin\rotatelogs.exe C:/hp/hpsmh/logs/error_log 5M 0 32 3224 rotatelogs.exe Command Line: C:\hp\hpsmh\bin\rotatelogs.exe C:/hp/hpsmh/logs/access_log 5M 0 32 3244 pvlsvr.exe Svcs: BackupExecDeviceMediaService Command Line: "C:\Program Files\Symantec\Backup Exec\pvlsvr.exe" 0 32 3256 hpsmhd.exe Command Line: C:\hp\hpsmh\bin\hpsmhd.exe -d C:/hp/hpsmh -f C:/hp/hpsmh/conf/smhpd.conf 0 32 3304 rotatelogs.exe Command Line: C:\hp\hpsmh\bin\rotatelogs.exe C:/hp/hpsmh/logs/error_log 5M 0 32 3312 rotatelogs.exe Command Line: C:\hp\hpsmh\bin\rotatelogs.exe C:/hp/hpsmh/logs/access_log 5M 0 32 4168 cpqnimgt.exe Svcs: CpqNicMgmt Command Line: C:\WINDOWS\system32\CPQNiMgt\cpqnimgt.exe 0 32 4200 cqmgserv.exe Svcs: CqMgServ Command Line: C:\WINDOWS\system32\CpqMgmt\cqmgserv\cqmgserv.exe 0 32 4224 cqmgstor.exe Svcs: CqMgStor Command Line: C:\WINDOWS\system32\CpqMgmt\cqmgstor\cqmgstor.exe 0 32 4248 SQLAGENT90.EXE Svcs: SQLAgent$SERVERNAME Command Line: "C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\SQLAGENT90.EXE" -i SERVERNAME 0 32 4340 svchost.exe Svcs: W3SVC Command Line: C:\WINDOWS\System32\svchost.exe -k iissvcs 0 32 4632 cqmghost.exe Svcs: CqMgHost Command Line: C:\WINDOWS\system32\CpqMgmt\cqmghost\cqmghost.exe 0 32 4672 wmiprvse.exe Command Line: C:\WINDOWS\system32\wbem\wmiprvse.exe 0 32 4828 beserver.exe Svcs: BackupExecRPCService Command Line: "C:\Program Files\Symantec\Backup Exec\beserver.exe" 0 32 5112 benetns.exe Svcs: BackupExecAgentBrowser Command Line: "C:\Program Files\Symantec\Backup Exec\benetns.exe" 0 32 5192 bengine.exe Svcs: BackupExecJobEngine Command Line: "C:\Program Files\Symantec\Backup Exec\bengine.exe" 0 32 5332 svchost.exe Svcs: TermService Command Line: C:\WINDOWS\System32\svchost.exe -k termsvcs 0 32 6088 wmiprvse.exe Command Line: C:\WINDOWS\system32\wbem\wmiprvse.exe 0 32 6000 svchost.exe Svcs: TapiSrv Command Line: C:\WINDOWS\System32\svchost.exe -k tapisrv 2 32 6352 csrss.exe Title: Command Line: C:\WINDOWS\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16 2 32 6376 winlogon.exe Command Line: winlogon.exe 2 32 6652 rdpclip.exe Title: RDPClip - Receive Thread Command Line: rdpclip 2 32 6732 explorer.exe Title: Program Manager Command Line: C:\WINDOWS\Explorer.EXE 2 32 6864 ccApp.exe Title: CCALERT Command Line: "C:\Program Files\Common Files\Symantec Shared\ccApp.exe" 2 32 6872 SmcGui.exe Title: Network Activity Command Line: \\.\pipe\SygateSecurityAgentR6334T202109 \\.\pipe\SygateSecurityAgentW26500T202109 2 32 6896 ctfmon.exe Title: Command Line: "C:\WINDOWS\system32\ctfmon.exe" 0 32 2492 logon.scr Command Line: logon.scr /s 2 32 2056 cmd.exe Title: C:\WINDOWS\system32\cmd.exe - cscript adplus.vbs -hang -pn sqlservr.exe -quiet -o c:\MDUMP Command Line: "C:\WINDOWS\system32\cmd.exe" 2 32 7944 taskmgr.exe Title: Windows Task Manager Command Line: taskmgr.exe 2 32 7896 SqlWb.exe Title: Microsoft SQL Server Management Studio Command Line: "C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\SqlWb.exe" 0 32 5680 dllhost.exe Svcs: COMSysApp Mts: System Application Command Line: C:\WINDOWS\system32\dllhost.exe /Processid:{02D4B3F1-FD88-11D1-960D-00805FC79235} 0 32 2284 DTExec.exe Command Line: /FILE "\\SVROne\SSIS.dtsx" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF /REPORTING E 2 32 6320 cscript.exe Title: Command Line: cscript adplus.vbs -hang -pn sqlservr.exe -quiet -o c:\MDUMP 2 32 6308 cmd.exe Title: C:\WINDOWS\system32\cmd.exe Command Line: "C:\WINDOWS\system32\cmd.exe" /c tlist.exe -v >"c:\MDUMP\Hang_Mode__Date_08-05-2008__Time_15-37-0707\Process_List.txt" 2 32 5240 tlist.exe Command Line: tlist.exe -v Edited August 5, 2008 by Bad boy Warrior
cluberti Posted August 5, 2008 Posted August 5, 2008 80000005 = an error has occurred (not really useful) and 0xa000000 = what is logged when the next admin user logs on after a machine is forcefully powered off and chooses a reason (any reason) from the unclean shutdown dialog box.Honestly, it'd be better if you could get a complete dump of the entire system the next time this occurs, because what we've got currently is not going to be enough to help you.
Bad boy Warrior Posted August 13, 2008 Author Posted August 13, 2008 When i created a full system dump i looked at the error codes and it seems all the error codes point to a manual dump being started (ifi understood it correctly):----- 32 bit Kernel Full Dump AnalysisDUMP_HEADER32:MajorVersion 0000000fMinorVersion 00000eceDirectoryTableBase 00039000PfnDataBase 81400000PsLoadedModuleList 808af9c8PsActiveProcessHead 808b5be8MachineImageType 0000014cNumberProcessors 00000008BugCheckCode 000000e2BugCheckParameter1 00000000BugCheckParameter2 00000000BugCheckParameter3 00000000BugCheckParameter4 00000000PaeEnabled 00000000KdDebuggerDataBlock 8089d3e0Physical Memory Description:Number of runs: 3 FileOffset Start Address Length 00001000 00001000 0009e000 0009f000 00100000 dfd56000 dfdf5000 dfe5e000 00001000Last Page: dfdf5000 dfe5e000KiProcessorBlock at 808aedc0 8 KiProcessorBlock entries: ffdff120 f7727120 f772f120 f7737120 f773f120 f7747120 f774f120 f7757120Windows XP Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatibleBuilt by: 3790.srv03_sp2_gdr.070304-2240Kernel base = 0x80800000 PsLoadedModuleList = 0x808af9c8Debug session time: Fri Aug 08 09:06:29 2008System Uptime: 0 days 0:12:27 start end module name80800000 80a7a000 nt Checksum: FFFFFFFF Timestamp: unavailable (FFFFFFFE)Unloaded modules:b81d9000 b81ee000 NAVENG.SYS Timestamp: Fri Aug 08 08:55:24 2008 (489BFBEC)b81ee000 b82be000 NAVEX15.SYS Timestamp: Fri Aug 08 08:55:24 2008 (489BFBEC)b81d9000 b81ee000 NAVENG.SYS Timestamp: Fri Aug 08 08:55:23 2008 (489BFBEB)b81ee000 b82be000 NAVEX15.SYS Timestamp: Fri Aug 08 08:55:23 2008 (489BFBEB)ba0e0000 ba0f5000 NAVENG.SYS Timestamp: Fri Aug 08 08:55:22 2008 (489BFBEA)ba0f5000 ba1c5000 NAVEX15.SYS Timestamp: Fri Aug 08 08:55:22 2008 (489BFBEA)baf50000 baf5e000 imapi.sys Timestamp: Fri Aug 08 08:54:25 2008 (489BFBB1)f7817000 f781f000 halfinch.sys Timestamp: Fri Aug 08 08:54:23 2008 (489BFBAF)f780f000 f7817000 Sfloppy.SYS Timestamp: Fri Aug 08 08:54:23 2008 (489BFBAF)f7607000 f7611000 Flpydisk.SYS Timestamp: Fri Aug 08 08:54:23 2008 (489BFBAF)f74f7000 f7504000 WDFLDR.SYS Timestamp: Fri Aug 08 08:54:10 2008 (489BFBA2)f723e000 f72a6000 bxvbdx.sys Timestamp: Fri Aug 08 08:54:10 2008 (489BFBA2)Finished dump checkWhen reading the reason why it shutdown from eventviewer i now have:Event Type: ErrorEvent Source: EventLogEvent Category: NoneEvent ID: 6008Date: 13/08/2008Time: 08:27:23User: N/AComputer: ELMI01Description:The previous system shutdown at 20:36:29 on 12/08/2008 was unexpected.For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.Data:0000: 000807d8 000c0002 00240014 0051001d0010: 000807d8 000c0002 00240013 0051001dThe data filed was in bytes but i clicked "words" to see if that makes it any helpful in resolving this. Or any other ideas if the log is of no help?Thanks
cluberti Posted August 13, 2008 Posted August 13, 2008 OK, you created a dump - can you put the .dmp file (compressed, of course) somewhere where it can be downloaded and analyzed?
Bad boy Warrior Posted August 13, 2008 Author Posted August 13, 2008 OK, you created a dump - can you put the .dmp file (compressed, of course) somewhere where it can be downloaded and analyzed?I could but its around 3.5GB. Not sure what Winrar will compress this to unless theres any other info i can give?Thanks
cluberti Posted August 13, 2008 Posted August 13, 2008 OK, you created a dump - can you put the .dmp file (compressed, of course) somewhere where it can be downloaded and analyzed?I could but its around 3.5GB. Not sure what Winrar will compress this to unless theres any other info i can give?ThanksProbably down to around 500 or so MB.
Mr Snrub Posted August 13, 2008 Posted August 13, 2008 Windows XP Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatibleBuilt by: 3790.srv03_sp2_gdr.070304-2240Kernel base = 0x80800000 PsLoadedModuleList = 0x808af9c8Debug session time: Fri Aug 08 09:06:29 2008System Uptime: 0 days 0:12:27Shooting from the hip time You're running the W2K3 SP2 (GDR) kernel - I don't know if your system or SQL is particularly registry-intensive, but you may be experiencing the symptom from http://support.microsoft.com/kb/935926 : "A Windows Server 2003-based computer stops responding when the registry is in heavy use"The latest (QFE) version of the kernel I could find (which includes the above hotfix and others) is available to the public here - http://support.microsoft.com/?id=954337 : "The computer could potentially have problems if DPC routines are executed at an IRQL that is greater than DISPATCH_LEVEL in Windows Server 2003"Without a dump this is of course pure speculation
Recommended Posts
Please sign in to comment
You will be able to leave a comment after signing in
Sign In Now