Rebasing System DLLs (from uSP2)

[Petr]

the point of rebasing is to avoid the fragmentation of the VIRTUAL Address space which is generated for every running process new and and only for the used adresses underlayed with physical (or paging disk memory).

example:

-free memory in this process: physical free + disk paging 500MByte

-free virtual memory 1.5 Gigybyte

-program try to allocate a chunk of 200Mbyte memory

-Does this work always?

looks like enough mem or? ... but to try to allocate an continous block of memory of 200MBytes CAN fail!

because (maybe) there is no contignous block of virtual mem of this size available, because of virtual memory fragmentation (perhaps caused by bad based DLLs).

I don't understand exactly.

With reference to MSRK:

does this mean that if the process tries to allocate 200 MB memory inside of its separate 4GB memory space, the Windows 98 system has to find contignous memory either in physical memory or in the swap file? And what dll rebasing could change on it?

Petr

[shaddam]

the point of rebasing is to avoid the fragmentation of the VIRTUAL Address space which is generated for every running process new and and only for the used adresses underlayed with physical (or paging disk memory).

example:

-free memory in this process: physical free + disk paging 500MByte

-free virtual memory 1.5 Gigybyte

-program try to allocate a chunk of 200Mbyte memory

-Does this work always?

looks like enough mem or? ... but to try to allocate an continous block of memory of 200MBytes CAN fail!

because (maybe) there is no contignous block of virtual mem of this size available, because of virtual memory fragmentation (perhaps caused by bad based DLLs).

I don't understand exactly.

With reference to MSRK:

does this mean that if the process tries to allocate 200 MB memory inside of its separate 4GB memory space, the Windows 98 system has to find contignous memory either in physical memory or in the swap file? And what dll rebasing could change on it?

Petr

hello petr,

no, windows has to find only contignous memory in the virtual space...(physical memory can be fragmented, no problem)

.... but finding a virtual memory chunk of 200Mbyte can be a problem if this space is divided by many DLLs (or other small allocated memory chunks)

DLL rebasing can move those small memory chunks to larger or (in best case) to one big memory chunk.

best case (unfragmented) :

- available virtual memory = contignous-in-one-block allocateable memory

worst case (fragmented) :

- available virtual memory >> contignous-in-one-block allocateable memory (typical factor 2-10)

Edited October 27, 2006 by shaddam

[LLXX]

The hardware can still remap virtual addresses - to a different area of physical memory, or even to the swap file. FYI, all the processors since the 386 supported up to 256 terabytes of virtual memory via this mechanism.

Indeed, there is fragmentation within the VM of each process, but since processes are not static, simply restarting them is enough to reorganise the virtual memory layout. Besides, if a process is doing enough alloc()s and free()s to fragment the virtual memory badly, whoever wrote the code is not paying attention to how memory allocation works.

One way to force the DLLs to be contiguous within memory, if that's what you intended, is to rebase all of them that have relocs to the same address. That way the OS will almost always have to relocate them somewhere else, and they'll usually be loaded contiguously.

...and the word is contiguous. Google shows that nearly 2^10 sites have it spelt wrong

Edited October 27, 2006 by LLXX

[oscardog]

The hardware can still remap virtual addresses - to a different area of physical memory, or even to the swap file. FYI, all the processors since the 386 supported up to 256 terabytes of virtual memory via this mechanism.

Indeed, there is fragmentation within the VM of each process, but since processes are not static, simply restarting them is enough to reorganise the virtual memory layout. Besides, if a process is doing enough alloc()s and free()s to fragment the virtual memory badly, whoever wrote the code is not paying attention to how memory allocation works.

One way to force the DLLs to be contiguous within memory, if that's what you intended, is to rebase all of them that have relocs to the same address. That way the OS will almost always have to relocate them somewhere else, and they'll usually be loaded contiguously.

...and the word is contiguous. Google shows that nearly 2^10 sites have it spelt wrong

By rebasing them all to the same base address would cause the dlls to be immediately be dumped to swap and result in that dll being made unavailable to share with anything else. Each subsequent app requiring the same dll would then have to load another one, again being made unavailable for sharing and so on and so on. Some of the times it is the C++ compilers error Borland,Watcom,LCC-win32 which all place the dll at 0x00400000 causing all these problems. Again as you say whoever wrote the code is not paying attention to how memory allocation works and are probably completely unaware concerning conflict issues. Rebasing the dlls properly can alleviate these problems and add some protection against buffer overflows etc from some exploits to come (this is the bit I am interested in)

[LLXX]

By rebasing them all to the same base address would cause the dlls to be immediately be dumped to swap and result in that dll being made unavailable to share with anything else.

No, that's why most DLLs have relocs. They can be relocated somewhere else, and usually that somewhere else is immediately next to the previously loaded one.

Of course, I'm talking about small lesser-used DLLs here. Rebasing the kernel and shell DLLs is just pointless, since their load addresses are fine as-is.

[shaddam]

intressting i never heard about a second hardware memory management unit which virtualize the virtual adress space again... can you please post some references to this?

all i know about is the software reloc by the OS whic got several disadavantages over correct basing.

but your second point is true... if we rebase all DLLs to the same address all should be based by the OS contiguously (sorry i'm not natural english speaker) ... with some performance disadavantages and some physical memory loss for the whole system. an intressting idea but only the second best solution over basing all to an perfect place.

The hardware can still remap virtual addresses - to a different area of physical memory, or even to the swap file. FYI, all the processors since the 386 supported up to 256 terabytes of virtual memory via this mechanism.

Indeed, there is fragmentation within the VM of each process, but since processes are not static, simply restarting them is enough to reorganise the virtual memory layout. Besides, if a process is doing enough alloc()s and free()s to fragment the virtual memory badly, whoever wrote the code is not paying attention to how memory allocation works.

One way to force the DLLs to be contiguous within memory, if that's what you intended, is to rebase all of them that have relocs to the same address. That way the OS will almost always have to relocate them somewhere else, and they'll usually be loaded contiguously.

...and the word is contiguous. Google shows that nearly 2^10 sites have it spelt wrong

[LLXX]

intressting i never heard about a second hardware memory management unit which virtualize the virtual adress space again... can you please post some references to this?

http://www.intel.com/design/pentium4/manuals/index_new.htm

[shaddam]

intressting i never heard about a second hardware memory management unit which virtualize the virtual adress space again... can you please post some references to this?

http://www.intel.com/design/pentium4/manuals/index_new.htm

maybe i don't understand your point but i can't find references for automatic memory shifting of virtual memory...

wikipedia explanation of virtual memory managment... as you can see the virtual memory chunks are contiguous (so don't seems to mapped again to another space) and alos fragmented... spaces between kernel32.dll and user32.dll

http://en.wikipedia.org/wiki/Virtual_address_space

From microsoft article:

"...Virtual memory fragmentation is a condition where virtual memory is available for a process, but none of the virtual memory blocks that are available are of a significant size."

From: http://support.microsoft.com/kb/325044

so...at least MS Windows can't reshape the virtual memory outfit of an running process. if the virtual addresspace is fragmented neither MMU or OS can do anything (without breaking a application which needs the granted addresses)