Rebasing System DLLs (from uSP2)

[shaddam]

hello

at work i have allocate often memorychunks of giant size (several 100 MByte to GigaByte), which often don't work even while the total amount of free memory seems way enough.

So i had to learn that there is something like virtual memory fragmentation. I explored this topic and found that there is an solvable reason for this on win32 system why the available memory and the maximum (on block) allocateable memory differs sometimes massive.

The reasons are often DLLs which are loaded in memoryregions which divide the available memory in 2 pieces instead of setting this DLLs on begin or end of this free region.

here an typical report from matlab memdump (win2000)

Startadress | size | free mem. after (with* it is allocatable)

...

<anonymous> 21610000 00010000 00000000

<anonymous> 21620000 00100000 418e0000*

C:\WINNT\system32\DDRAW.dll 63000000 00049000 00fb7000*

C:\Programme\MATLAB\bin\win32\MFC42.DLL 64000000 000f2000 052ae000*

C:\WINNT\system32\OPENGL32.dll 693a0000 000c7000 02d69000*

C:\WINNT\system32\MFC42LOC.DLL 6c1d0000 0000e000 02152000*

C:\WINNT\system32\INDICDLL.dll 6e330000 00006000 016ba000*

C:\WINNT\system32\GLU32.dll 6f9f0000 00020000 01d00000*

C:\WINNT\system32\COMCTL32.DLL 71710000 00084000 0107c000*

C:\WINNT\system32\DCIMAN32.dll 72810000 00006000 0277a000*

C:\WINNT\system32\WS2HELP.DLL 74f90000 00008000 00008000

C:\WINNT\system32\WS2_32.DLL 74fa0000 00014000 0000c000

C:\WINNT\system32\WSOCK32.dll 74fc0000 00009000 000f7000*

...

As you can see the loaded DLLs are not as near as possible together... the memory is fragmented.

But why the dlls want to reside in this special places? Because in the PE header of an DLL (or exe) a prefered

adress is defined (standard value = 10000000h).

The OS tries to load an DLL on this prefered place...but what happens when this place is already used by another DLL?

Then the OS has to do an dynamical reallocating which got some disadvantages.

First, loading is slower, and all adress calculations have to be recalculated in realtime on access. Second is that an dll which is loaded by several process (like several open explorer windows) has to be loaded several times in memory -> real memory loss. If the dll can live on the prefered place the system can 'map' the same loaded dll to several process (physical memory saved).

So, for an optimal behaving system the Dlls have be placed on optimal adresses.

Microsoft tried this for their dlls but it seems they do it not anymore consequently (especially for fixed files for win9x!).

the region which they tried to fill continously is 0x70000000 to 0x7FFFFFFF.

So why not try to do this again for our uSP2?

------------------------------------

Tools for Rebasing

-To get an overview over loaded DLLs (and reallocations and memory fragmentation), Sysinternals ProcessExplorer is very usefull.

http://www.sysinternals.com/Utilities/ProcessExplorer.html

-Cygwin rebasing tool (works for also for stripped files but not for MS files)

http://www.tishler.net/jason/software/rebase/

-For MS DLLs rebase.exe from MS Visualstudio (search the folder)

More Info

- Dr.dobbs articel on rebasing

http://www.ddj.com/184416272

-nice explanation of dll rebasing

http://www.codeproject.com/dll/rebase.asp

-load performance test

http://msdn.microsoft.com/msdnmag/issues/0500/hood/

-interessting link about security reasons for DLL rebasing

http://archive.cert.uni-stuttgart.de/archi...2/msg00018.html

PS:

Contras

- why not doing this for ALL System DLLs? because some 'bad' written software have hardcoded stuff like absolute jumps inside which need the original base address but this is some kind of hack, which could break this software also in other circumstances.

- rebasing conflicts possibly with the MS eula because it is some kind of binary hack(?)

Pros

- sligthly faster loading times for dlls

- sligthly faster dll functions

- slightly more physical memory

- much more virtual memory

- maleware which needs those known bases don't work anymore

Edited October 26, 2006 by shaddam

[jimmsta]

This sounds like an awesome idea, in terms of better memory management, especially on older, slower systems running little ram, and slow hard drives. In terms of finding what files were compiled with MSVS, we can use PEiD, which can determine the compiler from tags within the file.

Now, how would we figure out what can be rebased? I figure that the w9x kernel and all other files have already had some sort of optimizations like this applied to them, and that most system-level dll's cannot be rebased due to hard-coded dependencies within other system-level executables.

This is an interesting topic, and I'll be fooling around with this a bit, to see what works, and what doesn't. DLL's that are known to take up large chunks of memory or known to get fragmented a lot should be first on our list of things to do.

All I can say is, thanks for posting this bit of information. It looks quite interesting, and should provide hours of fun research for those of us that are interested.

[oscardog]

Take this with a pinch of salt but might it be that the api calls (stacks) are not being flushed causing multiple same dll calls as your program moves on resulting in memory issues , might be more than fragmentation.

[jimmsta]

Let's see what results we can get on a system that has the UnloadDLLs registry tweak applied... I suppose that was microsoft's way to 'fix' things... but disabled it to prevent extra crashing... or something.

[shaddam]

thank you for your replies and interest...

i also have to fool around to get some resonable results... perhaps a whitelist & blacklist could be usefull?

perhaps everyone can share his experience here...

WHITELIST (DLLs which are working rebased without problems/crashes/conflicts)

-------------------------------------------------------------------------------------------------

- ddraw.dll rebased succesfull (MS rebase.exe) without problems/conflicts/crashes (dx9)

- ?

BLACKLIST (DLLs which are needed by some aplications exacatly at that address)

-------------------------------------------------------------------------------------------------

- ?

- ?

- ?

[noguru]

Let's see what results we can get on a system that has the UnloadDLLs registry tweak applied... I suppose that was microsoft's way to 'fix' things... but disabled it to prevent extra crashing... or something.

I'm afraid that there a enough dll's running constantly to keep your memory defragmented. I suppose that because of this the so-called memory defragmenters won't work too.

[shaddam]

Let's see what results we can get on a system that has the UnloadDLLs registry tweak applied... I suppose that was microsoft's way to 'fix' things... but disabled it to prevent extra crashing... or something.

I'm afraid that there a enough dll's running constantly to keep your memory defragmented. I suppose that because of this the so-called memory defragmenters won't work too.

hello noguru,

even one running wrong placed DLL can half (in worst case) your maximum available in-one-block-allocatable memory.

a DLL keep loaded until ALL processes are killed which use them AND unload dll registry switch is active... so for one process the (virtual) memory region is fragmented by the used DLLs.

only after killing the last process which use this DLL the memory space is *maybe* defragmented (from this special DLL)

nice article about memoryfragmen. & dll rebasing

http://www.ittvis.com/services/techtip.asp?ttid=3346

MS explanation about the unload DLL registry switch

http://msdn.microsoft.com/library/default....g/debugging.asp

this site lists alwaysunload dll as 'Bad Optimization'

http://www.tweakhound.com/xp/xptweaks/supertweaks11.htm

[jimmsta]

Is there any way that the rebasing could take place on load-time? That is, we create some sort of kernel hook that looks for white-listed dll's, and attempts to allocate a contigiuos block of memory for that DLL to load itself in.

I think some sort of runtime patching would be more useful, especially when creating a whitelist of files that work properly with rebasing.

Then again, that search for a continguous block of memory may be slower than rebasing the file the normal way.

I guess we're just looking for ways to get around re-writing the memory managment subsystem for Windows 9x.

[oscardog]

"BLACKLIST (DLLs which are needed by some aplications exacatly at that address)"

Would any of the dlls have problems after rebasing, I thought the os would intercept any calls via pointers to the stack, making the app unaware.

Edited October 25, 2006 by oscardog

[jimmsta]

I assume that would be true, but I wonder if Microsoft hard-coded memory addresses for certain system processes - which would mean that some processes may expect data to be in a specific place in memory. I wouldn't doubt that there's some sort of protection against rebasing some of the critical system dll's.

But hey, why not try? There's always a good chance that all this is hogwash, and rebasing will work on all DLL's, no matter what.

[shaddam]

Found out that the "Citrix Presentation Server 4.0." do some automatic DLL Rebasing... exactly what we want to do.

-citrix explanation

http://support.citrix.com/article/CTX10602...archID=33095188

-virtual memory savings (per user)

http://64.233.183.104/search?q=cache:RCBH_...t=clnk&cd=1

[oscardog]

Found out that the "Citrix Presentation Server 4.0." do some automatic DLL Rebasing... exactly what we want to do.

-citrix explanation

http://support.citrix.com/article/CTX10602...archID=33095188

-virtual memory savings (per user)

http://64.233.183.104/search?q=cache:RCBH_...t=clnk&cd=1

"the default exclusion list contains all known applications and DLLs that have been problematic in the past. This default exclusion list is in the registry location mentioned above."

Thanks for the info very handy to know as jimmsta says it seems well worth a try, has anybody got a win2k/3 server to install the software and obtain the list so these known dlls can be built around

Edited October 26, 2006 by oscardog

[oscardog]

I came across this and wondered if it might be helpfull http://www.codeproject.com/dll/niftyloadlib.asp seems to work similar to how citrix rebases

[LLXX]

I would advice against this practice. To avoid memory fragmentation? That is hardly a performance issue as the paging/protected virtual mode of the i386+ processors manages memory mapping in hardware.

Besides, I'm quite sure there are hardcoded addresses in many of the DLLs, and some of them do not even have relocs and can't be loaded anywhere else without extensive editing.

Basically, the performance gain (if any) from doing this is negligible.

[shaddam]

hello llx,

thanks for your great works for the win98 community... but in this case you are not right.

the point is NOT memory fragmentation which is not existing because of the virtual memory managment capability of the processor (which you state correct) which maps physical memory chunks from everywhere (which don't have to be continous) to an continous block of virtualmemory together (which is underlayed by physical memory).

the point of rebasing is to avoid the fragmentation of the VIRTUAL Address space which is generated for every running process new and and only for the used adresses underlayed with physical (or paging disk memory).

example:

-free memory in this process: physical free + disk paging 500MByte

-free virtual memory 1.5 Gigybyte

-program try to allocate a chunk of 200Mbyte memory

-Does this work always?

looks like enough mem or? ... but to try to allocate an continous block of memory of 200MBytes CAN fail!

because (maybe) there is no contignous block of virtual mem of this size available, because of virtual memory fragmentation (perhaps caused by bad based DLLs).

I would advice against this practice. To avoid memory fragmentation? That is hardly a performance issue as the paging/protected virtual mode of the i386+ processors manages memory mapping in hardware.

Besides, I'm quite sure there are hardcoded addresses in many of the DLLs, and some of them do not even have relocs and can't be loaded anywhere else without extensive editing.

Basically, the performance gain (if any) from doing this is negligible.