WildBill

Never mind: I had a flash of insight and figured out what the problem was: my rewritten kernel32 was passing a starting address to calc.exe that was "incorrect" because calc was loading the original kernel32 when it started up. When I temporarily hardcoded the starting address it would have normally received, it started up just fine. This doesn't make the rewritten kernel32 ready for use (it still causes the VM to crash when I completely replace the original one), but WinExec should be at least somewhat sound now (though I haven't tested Win16 apps with it yet). Now to write more tests...

I think I'm going to need some help with my kernel32 rewrite. I have it passing 130 unit tests, but I can't seem to get WinExec working. I've loaded up most of the routines involved with debugging output and I can't find anything wrong, but it doesn't properly spawn the process. I can post the project sources if anyone would like to help tracking the problem down. The way I test it is to copy the new kernel32 to kernel32_new.dll, and have my test program perform tests against the normal kernel32 and the rewritten one and compare the results. My WinExec test attempts to spawn calc.exe from both of them. It works with the normal kernel32, but the rewritten one doesn't fully spawn the process. I get a running calc.exe in Task Manager but the window never appears and it seems to want to consume about 5% CPU until I kill it. I've been fighting with this for a couple of weeks and I'm stumped. Testing this way lets me protect my system since I don't have to replace kernel32, though so far I haven't tested any kernel32 routines that change anything--only the ones that read info or do things that don't affect the system.

We really need a new service pack...

I switched to a new mouse on my laptop and had to make a bunch of improvements to this tool. This should be useful for anyone who has a Synaptics driver for their laptop and would like it to automatically disable the touchpad and joystick when a mouse is plugged in (and re-enable them when the mouse is unplugged). With all the changes I was tempted to bump the version to 1.1, but left it at 1.0.2. Still, it's been improved quite a lot since the last version, and for me it's one of those little utilities I can't live without.

I finally have all of the required kernel32 functions written in C and I've started writing unit tests... AreFileApisANSI PASSED FreeEnvironmentStringsA PASSED FreeEnvironmentStringsW PASSED GetACP PASSED GetCommandLineA PASSED GetCommandLineW PASSED GetComputerNameA PASSED GetComputerNameW PASSED GetConsoleAliasExesLengthA PASSED GetConsoleAliasExesLengthW PASSED GetConsoleCP PASSED GetConsoleDisplayMode PASSED GetConsoleInputExeNameA PASSED GetConsoleInputExeNameW PASSED GetConsoleKeyboardLayoutNameA PASSED GetConsoleKeyboardLayoutNameW PASSED GetConsoleOutputCP PASSED GetConsoleTitleA PASSED GetConsoleTitleW PASSED GetConsoleWindow PASSED GetCurrentDirectoryA PASSED GetCurrentDirectoryW PASSED GetCurrentProcess PASSED GetCurrentProcessId PASSED GetCurrentThread PASSED GetCurrentThreadId PASSED GetDefaultSortkeySize PASSED GetDllDirectoryA PASSED GetDllDirectoryW PASSED GetEnvironmentStringsA PASSED GetEnvironmentStringsW PASSED GetLinguistLangSize PASSED GetLogicalDrives PASSED GetModuleFileNameA PASSED GetModuleFileNameW PASSED GetModuleHandleA PASSED GetModuleHandleW PASSED GetNumberOfConsoleFonts PASSED GetNumberOfConsoleMouseButtons PASSED GetOEMCP PASSED GetPriorityClass PASSED GetProcessAffinityMask PASSED GetProcessHandleCount PASSED GetProcessHeap PASSED GetProcessPriorityBoost PASSED GetProcessShutdownParameters PASSED GetProcessVersion PASSED GetProcessWorkingSetSize PASSED GetSystemDefaultLangID PASSED GetSystemDefaultLCID PASSED GetSystemDefaultUILanguage PASSED GetSystemPowerStatus PASSED GetSystemWindowsDirectoryA PASSED GetSystemWindowsDirectoryW PASSED GetTempPathA PASSED GetTempPathW PASSED GetThreadLocale PASSED GetThreadPriority PASSED GetTickCount PASSED GetUserDefaultLangID PASSED GetUserDefaultLCID PASSED GetUserDefaultUILanguage PASSED GetVersion PASSED GetVersionExA PASSED GetVersionExW PASSED GetWindowsDirectoryA PASSED GetWindowsDirectoryW PASSED GlobalMemoryStatus PASSED InterlockedCompareExchange PASSED InterlockedDecrement PASSED InterlockedExchange PASSED InterlockedExchangeAdd PASSED InterlockedIncrement PASSED IsDebuggerPresent PASSED lstrcatA PASSED lstrcatW PASSED lstrcpyA PASSED lstrcpyW PASSED lstrcpynA PASSED lstrcpynW PASSED lstrlenA PASSED lstrlenW PASSED NlsGetCacheUpdateCount PASSED Sleep PASSED TermsrvAppInstallMode PASSED

There are 310 functions in the XPSP2 version of exfat.sys. I don't have the time to fully reverse-engineer it, but here is what Ida spits out after a few days of trying to figure it out and plugging in whatever info I can find by Googling. It's at least a starting point if anyone wants to take a crack at rewriting it. One caveat: the C code that Ida generates never shows try...except or try...finally blocks so be sure to use the generated asm file for that logic.

Oh, no, its a lot of functions. With a lot of help from Google I've made a little sense of it, but it's still about 400 functions (a handful of which are rather large). The first problem with the sys I found is that one function uses SetFilterToken from ntoskrnl, which my kernel doesn't have. I suspect, though, that the biggest incompatibilities are in the data structures involved. It accesses some system-level data structures that tend to differ between XP and 2k (though I'd have to completely reverse-engineer the data structures to be sure). It might be possible to rewrite it as a 2k-compatible version, but it would involve quite a bit of effort. I haven't even looked at the other files.

Hmm that's not *too* bad, but it depends a whole lot on what's in it. If you can post it somewhere I can do a quick run through in Ida tomorrow and see what it spits out. I paid for the Ida decompiler, so I can generate (albeit very messy) C code. It won't be recompilable out of the box, but it's step 1, as it were. At least it might shed some light on some of the logic. For me, the two most painful parts of the rewriting process have been reverse-engineering the structures and then figuring out what the really large and complex routines are trying to do. I've figured out a ton of internal structures in rewriting kernel32, but who knows how much that might overlap with a driver. If someone wants to take a stab at rewriting it (or even just the easier parts), I can probably provide enough information to get them at least something useful to start with. One caveat, though, the generated C code isn't always correct. I can also generate .asm files and .map files that are also useful. Often I have to correlate the two when something looks fishy in the generated C code.

exFAT support might also be in win32k.sys...don't hold me to that, but I have a nagging feeling that I might have seen some support for it there. It might be worth a quick search. How large is exfat.sys? If it's really smsll it might be worth rewriting.

Thanks! It's become apparent to me that in the long run we've got to rewrite these in C if we're ever going to get compatibility past a certain point (I don't just want to match XP, I want to eventually surpass it). I'm well past the 95% point of starting to have my first rev of kernel32 rewritten, but all this ntoskrnl work is keeping me away from it. I ve amassed enough info from reverse-engineering ntoskrnl that I could rewrite large portions of it with little difficulty, but the files are so huge that a complete rewrite would likely be a 2-year proposition (as it stands I'm coming up on a year for kernel32). I'm happy to do it since it's my OS of choice and I use it every day, but I think I could use some advice on how to break some of the effort up. Besides, it's always going to be easier to rewrite and test small portions at a time, but I'm not sure how to do it). Csrsrv and basesrv were small enough that I could rewrite them all at once but the kernel files aren't). On a related note, I'm trying to assist someone with getting SEP (Symantec Endpoint Protection) working with the new kernel. It's resulting with STOP errors (IRQL not less or equal, happening in IoGetAttachedDevice), and I'm not all that knowledgeable enough with WinDbg that I might be enough of a help. I managed to generate an updated .DBG file with the correct symbols for the uniprocessor ntoskrnl.exe, but that isn't helping all that much (and I can provide accurate .MAP files using a customized Ida Pro script I use). Would anyone be able to help diagnose this? Compatibility is of course the primary goal of extending the kernel, but it's causing me to put the kernel32 rewrite on hold. One of the things I think I need to add is an Ida script to generate .DBG files in addition to .MAP files (the built-in Ida function generates incorrect .MAP files, which is why I use my own script). If I can find the source to map2dbg or if someone can point me to it I should be able to add it to my Ida script. I also need to write an Ida script that can import a .MAP file and update the Ida database. That would be huge. If there are any Ida users who woukd like to use my .idc scripts just say the word. For me they've been critical to getting this far,

I've got a new v13 of KB2393802 up, which fixes one bug and adds a bunch of new kernel functions: ntdll.dll Bugfixes: RtlRandomEx -- Fixed a bug that would cause an exception (and would return an incorrect result anyway) New additions: NtOpenProcessTokenEx NtOpenThreadTokenEx ZwOpenProcessTokenEx ZwOpenThreadTokenEx ntoskrnl/ntkrnlpa/ntkrnlmp/ntkrpamp.exe New additions: RtlHashUnicodeString RtlRandomEx RtlInitializeGenericTableAvl RtlInsertElementGenericTableFullAvl RtlIsGenericTableEmptyAvl RtlGetElementGenericTableAvl RtlNumberGenericTableElementsAvl RtlInsertElementGenericTableAvl RtlDeleteElementGenericTableAvl RtlLookupElementGenericTableFullAvl RtlEnumerateGenericTableLikeADirectory RtlLookupElementGenericTableAvl RtlEnumerateGenericTableWithoutSplayingAvl RtlEnumerateGenericTableAvl RtlIpv4StringToAddressW RtlIpv4StringToAddressExW RtlIpv4AddressToStringW RtlIpv4AddressToStringExW RtlIpv4StringToAddressA RtlIpv4StringToAddressExA RtlIpv4AddressToStringA RtlIpv4AddressToStringExA RtlIpv6AddressToStringW RtlIpv6AddressToStringExW RtlIpv6AddressToStringA RtlIpv6AddressToStringExA RtlIpv6StringToAddressA RtlIpv6StringToAddressExA RtlIpv6StringToAddressW RtlIpv6StringToAddressExW ExGetCurrentProcessorCpuUsage ExGetCurrentProcessorCounts ExfInterlockedCompareExchange64 FsRtlIncrementCcFastReadNoWait FsRtlIncrementCcFastReadNotPossible FsRtlIncrementCcFastReadResourceMiss FsRtlIncrementCcFastReadWait Existing routines added to the export table: ExVerifySuite HalExamineMBR

http://www.mediafire.com/download.php?b9n29hdqwiw3hyu It's also in the post-EOL updates list...

Just posted KB2508429 v9. This should fix the problem

KB2508429 v9 is up, which should fix a STOP error when copying files over a network share...

Ah. Found it. I missed an instruction. I'll post an update ASAP.

Going to look at in Ida right now...

I've got a new installment of MS11-011 (KB2393802) posted (v12). By request, it adds a number of kernel-level functions: ntoskrnl/ntkrnlpa/ntkrnlmp/ntkrpamp.exe KeAreApcsDisabled IoQueryFileDosDeviceName MmProtectMdlSystemAddress KeQueryActiveProcessorCount PsDereferenceImpersonationToken PsDereferencePrimaryToken NtOpenProcessTokenEx NtOpenThreadTokenEx ZwOpenProcessTokenEx ZwOpenThreadTokenEx CcMdlWriteAbort Enjoy...

I've posted Windows2000-KB2508429-v8-x86-ENU.exe on the main list. It adds MiniDumpWriteDump to dbghelp.dll. It doesn't actually write the minidump, rather ti sets the E_FAIL error code and returns false just as the real one would do if it fails. However, it will let you run Star Ruler I've almost finished the draft cut of rewriting kernel32 -- 815 exported routines are in our of a total of 902, but I won't need that many before I can start testing and debugging it.

Well, so far I haven't been able to track down what's corrupting the heap. I think the best strategy is to finish the kernel32 rewrite since I'm so close to the end, and then perhaps move on to rewriting ntdll (which is smaller). That way I can try to put in better heap corruption detection and maybe find out what's going on.

There's something screwy going on...I backed up all the way to v3 and I still get the same occasional errors when accessing a network share from within a VM. I then tried a clean install of 2kSP4 and it still happens. I wonder if it's a VM thing. I'm using Virtual PC 2007.

I've been crawling through the code, double-checking and triple-checking everything, and I can't find anything wrong with the code. I tried backing some changes out and eventually replaced v9 ntdll with v8 ntdll and I still see occasional problems in a VM (though never on real hardware). From looking at the exceptions it looks like something is corrupting the heap, and subsequent heap operations are throwing exceptions. Mixing v8 ntdll with v9 kernel definitely isn't preventing the problem. What happens if you try a pure v8 install on a VM?

Now that's interesting. I'm seeing the same thing in my VM, but not on real hardware. It must date back to v9, since the ntdll in v9 and v10 are the same. v9 was where I added a bunch of SxS API's to ntdll. I guess I'll have to look at it closely to see if I broke anything.

Really? There were only two extremely minor changes. One was to correct the error code that blackwingcat pointed out and the other was to fix a bug when initializing a process -- the location of a particular field is different in 2k vs. XP. The thing is, though, the code that's affected should never actually run since it only gets invoked if a process was using an activation context, and that functionality isn't active in my kernel yet. I'm not having any problems here...can you provide any more info?

I've got a couple of updates posted: Windows2000-KB2393802-v11-x86-ENU.exe - A couple of ntdll bugfixes, and it should now be friendlier for slipstreaming. Windows2000-KB2508429-v7-x86-ENU.exe - Some new API functions added: dnsapi.dll NetInfo_Free DnsFree DnsFreeConfigStructure iphlpapi.dll GetTcpStatsFromStackEx GetIpStatsFromStackEx GetUdpStatsFromStackEx GetIcmpStatsFromStackEx GetTcpExTable2FromStack GetUdpExTable2FromStack GetExtendedTcpTable GetExtendedUdpTable GetBestInterfaceEx The 2k iphlpapi only supports ipv4, so while the calls above will work for ipv4, they'll properly report an error code if called for ipv6. My kernel32.dll rewrite is still proceeding apace...765 exported routines and counting.

I tried it once and it refused to let any fonts show up in the Control Panel. I haven't tried it since...