Cannot boot DC (ntds.dit)

[Bad boy Warrior]

For some strange reason i restarted our server and i couldn't boot!! I received a message few seconds into booting:

LSASS.EXE - System Error, security accounts manager initialization failed because of the following error: Directory Services cannot start. Error status 0xc00002e1.

Please click OK to shutdown this system and reboot into directory services restore mode, check the event log for more detailed information.

Ok now i get into the system via restore mode and run ntdsutil which gives me an error when i run the command ntdsutil "sem d a" "go f". The error i receive is Failed with jet error 501. Reading about this some say its a corrupt database and i may need to restore it from backup????

Problem is the C drive is never backed up only user files are (im guessing the backup is referring to C:\WINDOWS\NTDS\ntds.dit) . So before i go any further does anyone have any ideas/advice on what i could try??

Thanks

[cluberti]

Is this your only domain controller?

[Bad boy Warrior]

Is this your only domain controller?

Unfortunately......yes

[cluberti]

Tell me you have a backup... otherwise, you're in hot water. You can try running

ESENTUTL /g "<path>\NTDS.dit" /!10240 /8 /v /x /o

and then run an integrity check on the files, and a semantic database analysis, which should hopefully work - if they do, you should be able to reboot the DC properly, but there will be problems with the DC's database and you may still have lingering issues. Honestly, if that works, consider yourself lucky - bring up a new domain controller, make it a global catalog server, move the FSMO roles onto it, and dcpromo this failed server down to a member server. Then, rebuild it and add it back to the domain and dcpromo it to a DC/GC server as well, and LEAVE THEM BOTH UP AND RUNNING .

Never, ever, EVER have a single DC environment unless you are running SBS - and even then, always, ALWAYS back up your DC. Here's the disaster recovery powerpoint - read it

http://download.microsoft.com/download/0/3...AD_Disaster.ppt

[Bad boy Warrior]

No unfortunately the only backup i have is user files and nothing on the C drive of the server

[cluberti]

Well, hope eseutil works then - otherwise, you're starting over.

[Bad boy Warrior]

Tell me you have a backup... otherwise, you're in hot water. You can try running

ESENTUTL /g "<path>\NTDS.dit" /!10240 /8 /v /x /o

and then run an integrity check on the files, and a semantic database analysis, which should hopefully work - if they do, you should be able to reboot the DC properly, but there will be problems with the DC's database and you may still have lingering issues. Honestly, if that works, consider yourself lucky - bring up a new domain controller, make it a global catalog server, move the FSMO roles onto it, and dcpromo this failed server down to a member server. Then, rebuild it and add it back to the domain and dcpromo it to a DC/GC server as well, and LEAVE THEM BOTH UP AND RUNNING .

Never, ever, EVER have a single DC environment unless you are running SBS - and even then, always, ALWAYS back up your DC.

1. I did read about that so ill try those tricks tonight.....

2. What kind of problems could i expect assuming this works?

3. How do i move FSMO roles?

4. Overall should i backup the NTDS.dit file on our seperate SBS server to ensure this problem doesnt occur (this is a seperate server that i think ill take more precaution with)?

5. What could have caused this problem?

Finally if the above doesnt work am i right in saying that ill have to rebuild the server? If so can i backup the GPOs and import them if worse comes worse?

Thanks

Edited January 22, 2008 by Bad boy Warrior

[cluberti]

2. What kind of problems could i expect assuming this works?

Data will likely be missing from your domain - not sure what, but esentutil is usually able to "fix" database corruption, but does so by discarding corrupted data in most cases. You could be missing user accounts, unable to start services, etc. Not only that, but once you have to resort to esentutil to "fix" a corrupted AD, it's no longer supported by Microsoft as it could be tainted (due to corruption) in any number of ways. Again, in the event esentutil allows you to boot normally, consider the installation "broken" from then on until you can bring up a new DC as a global catalog server, move the FSMO roles, replicate a few times, and dcpromo the old DC down. Once you've done that, go over your AD with a fine-toothed comb and make sure everything (and I do mean everything) works. This would also be the time to backup your GPOs with GPMC, make a backup of your AD, etc.

3. How do i move FSMO roles?

http://www.petri.co.il/transferring_fsmo_roles.htm

4. Overall should i backup the NTDS.dit file on our seperate SBS server to ensure this problem doesnt occur (this is a seperate server that i think ill take more precaution with)?

No, backing up the .dit file is unnecessary. Follow the backup strategies on technet instead:

http://technet.microsoft.com/en-us/library/bb727048.aspx

5. What could have caused this problem?

Usually file system corruption, or the writeback cache on the raid controller wasn't written to disk successfully, or your antivirus product scanned the file and corrupted it, etc.

Finally if the above doesnt work am i right in saying that ill have to rebuild the server? If so can i backup the GPOs and import them if worse comes worse?

Too late for that - unless you've already backed them up with GPMC, you'll find they may or may not work on the new DC. You can try, but don't expect it to work - the GUIDs for everything in your new domain won't match the old, and it will likely simply ignore the old policies.

[Bad boy Warrior]

2. What kind of problems could i expect assuming this works?

Data will likely be missing from your domain - not sure what, but esentutil is usually able to "fix" database corruption, but does so by discarding corrupted data in most cases. You could be missing user accounts, unable to start services, etc. Not only that, but once you have to resort to esentutil to "fix" a corrupted AD, it's no longer supported by Microsoft as it could be tainted (due to corruption) in any number of ways. Again, in the event esentutil allows you to boot normally, consider the installation "broken" from then on until you can bring up a new DC as a global catalog server, move the FSMO roles, replicate a few times, and dcpromo the old DC down. Once you've done that, go over your AD with a fine-toothed comb and make sure everything (and I do mean everything) works. This would also be the time to backup your GPOs with GPMC, make a backup of your AD, etc.

3. How do i move FSMO roles?

http://www.petri.co.il/transferring_fsmo_roles.htm

4. Overall should i backup the NTDS.dit file on our seperate SBS server to ensure this problem doesnt occur (this is a seperate server that i think ill take more precaution with)?

No, backing up the .dit file is unnecessary. Follow the backup strategies on technet instead:

http://technet.microsoft.com/en-us/library/bb727048.aspx

5. What could have caused this problem?

Usually file system corruption, or the writeback cache on the raid controller wasn't written to disk successfully, or your antivirus product scanned the file and corrupted it, etc.

Finally if the above doesnt work am i right in saying that ill have to rebuild the server? If so can i backup the GPOs and import them if worse comes worse?

Too late for that - unless you've already backed them up with GPMC, you'll find they may or may not work on the new DC. You can try, but don't expect it to work - the GUIDs for everything in your new domain won't match the old, and it will likely simply ignore the old policies.

Ill have reinstall the whole server.....and to add to my missery our Server CD seems to be missing. Although i have the product key. Is it possible for me to download a trial version of Enterprise edition and use my product key? If i remember correctly? or am i going to have further problems?

[cluberti]

Is it possible for me to download a trial version of Enterprise edition and use my product key? If i remember correctly? or am i going to have further problems?

No, in fact trial versions won't accept retail or OEM product keys, at all. If you need media you can contact MS and fax them the COA, and they'll ship you a CD.

Is that a retail key, or an OEM key?

[Bad boy Warrior]

Is that a retail key, or an OEM key?

Retail product key but i have a OEM disk with OEM licenses (which of course i cant use) but could i use this disk and retail prod key or am i wasting time?

[cluberti]

Retail product key but i have a OEM disk with OEM licenses (which of course i cant use) but could i use this disk and retail prod key or am i wasting time?

OEM disk and retail key do not work - you must have a retail CD to use a retail key, and an OEM key to install an OEM OS. I'd contact Microsoft asap to get media.

[Stoic Joker]

Look at this as a golden opportunity to throughly document you domains configuration.

I maintain a 60+ page word document with complete details of our network configuration. Starting with high-level stuff Model, serial, and service tag numbers for all servers. A break down of which servers host which services, databases, etc. Partition sizes and a rough list of what's on them.

Basically anything that might come up as a show stopper at 2am during a bare metal rebuild is all contained in that documentation. I take screen shots of some of the more critical & intricate configurations for clarity and embed them in the document. (The title at the top of the document is Don't Panic! ... and yes it's in green letters )

Do not store the documentation for the server, on the server, because if you ever really need it ... you won't be able to get to it. I keep one copy on my WS which is backedup offsite seperately from the normal offsite backups we do for company data. I also give a printed copy to the company's owner to keep in her files just in case I get hit by a bus ... and they decide to continue opperating without me.