Jump to content

The 16bit heaps expander thread


Recommended Posts

All right guys, anyone able to make some patch (free or commercial) to transform those 16bit GDI and User segments into significantly bigger segments so that it is possible to run really many apps at once ?

Dencorso, if you happen to read this, could you please upload the pdf you made of the older thread that got deleted by mistake a year ago or so, if you still have it ?

For the life of me I can't find it anymore on my HDs and I think it would be nice if it could be uploaded because there is quite a good deal of serious discussion about the topic in it if I recall correctly.

Edited by eidenk
Link to comment
Share on other sites


My bottleneck when using Windows ME are the resources.

I have nearly 2GB of RAM. I can run scores of processes at once without stabilty issues. All this works very

well just only sometimes I am stuck for lack of resources because I open too many apps at once or too

many plugins in certain audio applications I use.

So that I am after a patch that could significantly increase the size of the 32bit segments of the resources.

I agree with eidenk: the resources problem (running too low on them too fast) is IMHO the major limitation

faced by us users of Win 9x/ME, and sure is worth looking at more closely. AFAIK, they are implemented

respectively in GDI.EXE/GDI32.DLL and USER.EXE/USER32.DLL. Some useful info about them is available

in now classic "Windows 95 System Programming Secrets" by Matt Pietrek.

More information of Win9x system resources and its limitations here:

http://www.helpwithwindows.com/techfiles/win-resources.html

and below:

Why the memory problem in Windows 95, 98 and Me isn't fixable

Copyright © 2001, Al Fasoldt

Copyright © 2001, The Syracuse Newspapers

The standard versions of Windows have a big problem handling memory. This is hardly news to most of

us who use Windows, but there's a hidden side to the problem that makes it worse than it might seem.

Even people who are otherwise knowledgeable about Windows miss the significance of the problem. The

basic flaw is bad enough -- Windows has only a tiny area of memory to keep track of what's going on --

but the hidden aspect is that the problem can't be fixed.

Crazy as it seems, this fatal flaw in Windows 95, Windows 98 and Windows Me is never going to go

away. No software utility can fix it. Neither MemTurbo nor RAM Idle, two of the most popular programs

that claim to fix Windows memory handling, do anything to fix the problem. Nothing will.

Keep that in mind the next time you see claims about programs that supposedly fix problems with

Windows. A software program can't fix this Windows memory flaw any more than a set of racing tires can

turn a Hyundai into a Ferrari. The design of Windows makes a fix impossible. Only a complete redesign of

Windows 95, Windows 98 and Windows Me would work -- and, in fact, that's what Microsoft is doing for

the next home version of Windows, which it plans to introduce later this year. That version is Windows XP.

Windows 95, Windows 98 and Windows Me are like big cars fitted with tiny gas tanks. They will run fine

for a while and then sputter to a stop when they use up their allotment of memory.

The memory I'm referring to isn't the standard kind of memory that all computers need to function properly. It's a special area of memory unique to Windows. It's called "resource" memory, where Windows

keeps its pointers to what's going on.

When that memory area gets full, Windows loses control. Your mouse might stoop responding or you'll find

that you can't close a window on your screen. You'll be stymied just trying to close programs, and your

Windows PC will freeze. You'll lose what you were working on.

Ready for the wacko part of this? This storage area is so small, so improbably tiny, that it's not even

expressed in megabytes. Your PC's hard drive probably has hundreds and hundreds of megabytes of

storage. (A megabyte is 1 million bytes, and a byte is the smallest unit of storage.) If it's a modern

computer, your PC's actual memory probably reaches all the way to 64 megabytes or more.

It sure would be great if Windows knew what to do with 64 megabytes for its resources. But let's not be so

greedy. It would be just fine if it knew how to deal with 32 megabytes. Or with 16 megabytes. Heck, I'd

settle for 8 megs. Or even 1.

But even 1 megabyte is out of the reach of Windows when it's dealing with resources. Want to hear the

really bad news about this resource memory? Windows 95, Windows 98 and Windows Me can't set aside

any more than 64 kilobytes for this vital storage. No matter how much memory your PC has -- and most

new ones have at least 64 megabytes of memory -- all that Windows can use for resources is 64 kilobytes

(usually called "64K").

This means Windows can only use one-tenth of 1 percent of the typical memory in a modern PC for the allimportant

function of keeping track of what's going on.

This is the bad news that seems to follow Windows users around like a toothache. I get dozens of letters a

week from Windows users looking for ways to keep their Windows 95, Windows 98 or Windows Me

computers from running out of memory and crashing. They often ask if memory-enhancing programs they

see on the Internet actually fix the problem.

The answer is no. The 64K limit in Windows 95, 98 and Me is a barrier that can't be taken down. No

program can change this. Adding more regular memory (adding RAM, in other words) won't fix it, either.

Rebooting (shutting down and starting up again) can help by clearing out resource memory. When resource

memory runs low again, reboot again.

Apart from rebooting, which hardly counts as a "fix," there is no way to cure this flaw in these versions of

Windows. If you want to run a version of Windows that handles memory properly, you have two current

choices -- Windows NT, an older version that is about to be retired, or Windows 2000, a new version I've

raved about in print and on TV. Or you can wait for Windows XP later this year.

With the introduction of Windows XP, which, because of Microsoft's nearly total monopoly on PC operating

systems, will show up on practically every new PC sold by the end of the year, Microsoft will at last be

making an advertised version of Windows that is reliable and able to keep running for weeks without the

need for rebooting. If you're not drawn to the Mac or to Linux through your frustrations with Windows 95,

Windows 98 and Windows Me, and if you're scared away by Microsoft's refusal to tell you that Windows

2000 actually exists, you will want to upgrade to Windows XP as quickly as possible.

galahs, dencorso, eidenk, rloew

From what I understand of the problem, in order to make Win 95-98-ME able to run (backward compatible with)

16-bit Win 3.X programs, they need to be able to operate in 16-bit mode. Thus the files, User.exe, User32.dll,

GDI.exe, and GDI32.dll limit the 16-bit system resource size to 64 kbyte. Likewise, there are three 32-bit system

resource limitations, 2 Mbyte each per this article (click "About System Resources" at the bottom):

http://www.mvps.org/serenitymacros/repair.html

Some more reading on the subject:

http://www.aumha.org/win4/a/resource.htm

After reviewing the various documents and web sites relating to "resources", I have come to the following conclusions:

The two 16-Bit 64K Resources are the most serious limitation, but are probably not expandable without

breaking many programs. I suspect that 16-Bit pointers may be used by various programs, all of which would

need to be fixed. Tracking down all of the programs that would need to be patched could be very tedious, and

doing so for third-party programs would be prohibitive.

The three 32-Bit 2MB Resources probably can be expanded more easily as long as the programs that use it

are not hard coded to expect 2MB. All of the references, that I have read, insist that these resources do not get

depleted in any reasonable use. I am not sure how useful expanding them would be.

I have as yet to locate the actual owner of these resources, but I currently suspect KERNEL32.DLL maintains

them.

Yes, they are THE question! What I had been thinking on, for a long time now, would be more of a workaround than a fix proper: I imagined a ring 0 program, say, srfix.vxd, that took a snapshot of the 16-bit USER stack, just after boot and saved it somewhere. Then, when the user resources get too low, the user closes all running programs, except those that were running just after boot and triggers a hotkey, causing srfix.vxd to restore the saved snapshot, thus releasing any orfaned resources lost due to leaks related causes. Ideally, the initial snapshot taking should be automatic, but as it's really not obvious in which order startup programs will load,perhaps that too would need to be an user triggered event. And srfix.vxd might do it independently also for GDI. I'm fully aware that this is not more than a very brute force workaround, but it may be a start. Please let me know what you think of this.
Thanks for the input everybody :thumbup , I suppose that this is the real root of the system resource problem.

Note that in two bytes there are ((2 raised to the 8th power) squared =) 65536 possible combinations, thus there is only a 64K 16-bit heap size.

"...The resource table is essentially a big list of information about all the resources that are in memory at any

given time. So if an application tells Windows to load a resource, Windows finds an empty spot in this resource

table, and fills it in with the information about the resource that was just loaded. Now, instead of giving the

application a four-byte pointer to the resource, Windows can just tell the application where the resource is in

the table. If I tell Windows to load a window, and that window winds up taking the 383rd slot in the resource

table, Windows will tell me "Okay, I've loaded the resource, and it's #383." Since these 'index numbers' are

much smaller numbers than memory addresses, under this scheme, a resource's number can be stored in

only two bytes instead of four; when you only have a few megabytes of memory to work with, and lots of

resources being used, that's a huge improvement.

There's a problem with this scheme. There's only so many different possible values that you can store in a

certain number of bytes of computer memory, just like there's only so many different numbers you can write

down if you aren't allowed to use more than a certain number of digits. If you have four bytes of memory to

work with, you can store billions of different possible values in those four bytes. But if you only have two bytes,

there's only 65536 different numbers that you can store in those two bytes. So if you use two-byte numbers as

your resource identifiers, you can't have more than 65536 resources loaded into memory at one time; if you

loaded more than that, there'd be no way for programs to tell them apart. But on the computers of the day,

there'd be no way to fit more than a few thousand resources into memory at one time anyway. So this

limitation wasn't seen as being a problem, and the Windows designers went ahead and used the resource

table and two-byte resource identifiers.

Now, we leap ahead to the present day. Memory is incredibly cheap; the memory savings from using two-byte

resource numbers instead of four-byte pointers simply aren't significant anymore. There'd be more than

enough memory to hold hundreds of thousands of resources in memory at one time. But there's still only

65,536 different possible resource identifiers; so only that many resources can be loaded into memory at once.

Beyond that, you're out of resources, no matter how much memory you have left."

http://www2.whidbey.com/djdenham/Window_memory.htm

Dear Dencorso,

In addition to the issues you have already mentioned there are additional problems with this approach. I have

listed some of the more apparent ones below:

1. This approach does not help when you are in the middle of something and your resources run out. You

need to terminate everything before you can use

the Hotkey. You only avoid having to reboot, which would probably be a good idea by this time to clear out

other waste.

2. There is no guarantee that you can terminate ALL processes, or that some processes started after startup

must continue to run for proper operation afterwards.

3. When resources run out, the computer could be locked up. preventing you from using the Hotkey or

executing the srfix.vxd.

As long as the resources discussion is still here, i'd like to come in with some thoughts, if I may. Please note

I'm no programmer so I may be talking nonsense, but hopefully some good idea could be hooked out of this.

Once the owner/maintainer of the resources is found, there could be added a hook that would maintain

multiple 16-bit heaps and a list of running applications, so when certain application requests a certain

resource, the hook could direct it to the correct heap.

Say 'application #5 requests resource #172' - the hook would know that the respective resource would be

found in heap #4, so it would redirect the request to the respective heap.

With multiple 16-bit heaps, I believe there would be no more resource problems. But what do I know... maybe

I'm far off. My apologies if so. :blushing:

Some Resources such as Windows can be accessed from more than one application so selecting Resource

heaps by application will not work.

A Resource indexing approach is a possibility I am considering. It will only work if all routines that use

Resource Handles to access Resource Data directly can be patched.

Note: all posts between Oct 19 2007 and Nov 15 2007 did not refer to resources.

Here is a nice article regarding Windows resources:

http://apptools.com/rants/resources.php

Under Windows 3.x, each of these two areas of memory was limited to 64K. All running applications shared that 64K for User resources and 64K for GDI. Needless to say, that created a huge bottleneck.

With the introduction of Windows 95 and continuing through Windows 98 and Windows Me, that 64K was increased dramatically and each of the two areas was further subdivided as follows:

- The 16-bit User heap (64K).

- The 32-bit User window heap (2MB).

- The 32-bit User menu heap (2MB).

- The 16-bit GDI heap (64K).

- The 32-bit GDI heap (2MB).

However, each one of these five memory sections is still fixed in size due to the Windows 9x/Me architecture, and cannot be increased, regardless of the amount of physical memory (RAM) installed on the machine. Modern applications demand more and more of the system. The more controls an application creates and the more files it opens, the more stress is placed on the operating system

Roman

If of any help regarding patching user and gdi to overcome system resource limitations I notice that the GDT (global descriptor table) is used mainly for holding descriptor entries of operating system segments. In which the descriptor fields Base points to the starting location in the 4GB linear address space, D/B segment size bit, when the bit is set the processor assumes a 32bit segment (here perhaps allows the modifying of the 16bit heaps to 32bit) along with LIMIT segment limit (20 bits) which defines the size of the segment dependant upon Granularity bit being set or clear. The descriptor type bit and segment type should hopefully already be set as needed

If the 16 bit heaps are changed to 32 bit and allowed 2mb in seqment size the rest of the global descriptor table would then need its existing entries moved 2mb -64kb to accomadate this. Perhaps rebasing of some files will be required to conform to the new table

Once upon a time, when Windows meant either Win 3.1 or Wfw 3.11, there was Helix Hurricane...

It had a "Heap Expander" that really did manage to expand the system's resources. It comprised two files: a 5.3KB VxD called HEAPX.386, and a 38.1KB NE driver, HEAPX.DRV... Helix got bought by McAfee (NAI, at that point) and never developed a Win 9x version for "Heap Expander", which was incompatible with Win 9x, because of the serious differences in implementation of GDI and USER heaps between Win 3.1 and Win 9x (described some posts above).

Be as it may, perhaps in-depth study of HEAPX.* might lend some interesting ideas on how to overcome the resources problem...

Edited by dencorso
Link to comment
Share on other sites

BTW, there is a tool for making sense of the resources percentages. It's called heap walker and it is part of Print99 on this page :

http://www.microsoft.com/downloads/details...;displaylang=en

A different version can be found here :

http://download.microsoft.com/download/pla...-us/newheap.exe

This might also be of interest :

http://msdn.microsoft.com/msdnmag/issues/01/03/leaks/

I could not compile the programs successfully with VC6 but I am probably missing something. Haven't looked into it more than that.

Thanks for the links, eidenk!

Let me add some more:

http://support.microsoft.com/kb/97759

and http://support.microsoft.com/kb/136776: Problems Using Helix Hurricane with Windows 95/98, where M$ sugests removing Hurricane altogether as a solution because... "Hurricane and its utilities are incompatible with Windows" 9x. Even so, it should be read.

So you think there isn't a more recent version of heap expander than the one you posted ?

With heap walker I have been able to see that the percentages that appear on the system resource meter are a mesurement of the 16bit segments only apparently. The 32bit segment don't appear to be taken into account and as far as I have seen are barely used at all.

Another simple resource viewer :

http://www.geocities.com/the_real_sz/misc/bear_.htm

Those might also be of some interest :

http://www.japheth.de/Download/TaskList.ZIP

http://www.japheth.de/Download/WinInfo.ZIP

I think there is a very thin chance of a further version in Hurricane 2.03, but I do honestly doubt it. See: http://web.archive.org/web/19971212004348/...ne_updates.html. The file I posted came with Hurricane v. 2.02 (a.k.a. Helix Hurricane '95). There was a v. 2.03 (later a.k.a. McAfee Hurricane 98). I had moved on to Win 9x by then, and didn't care to get it, 'cause I knew they just didn't mix. Helix stopped development of Heap Expander in 1996, didn't port it to Win 95 and was sold in 97/98 to McAffe, that discontinued Hurricane altogether in 2000. That's all I know and my best guesses. I do miss HEAPX to this day.

PS: All our problem does lie with the 16-bit heaps. The 32-bit heaps are large enough. That's why I believe it should be possible to adapt/recreate HEAPX for 9x, to manage the 16-bit resources, letting the 32-bit resources alone: it's less than HEAPX needed to do under 3.1, not more...

You never did run it under 9x yourself then ? The MS KB you posted shows that it was intended to run under Win 95 and 98 doesnt'it ? It would be worth trying maybe.

As Hurricane and Nuts and Bolts 98 appear very similar in appearance, I've just had a look into the latter but no luck for a heapx or similar in there.

Seems like both products were from Helix originally who sold them to McAffee or something.

Helix itself says "win 3.xx only" (see the manual pages I included in the .7z) and, while back then there were no forums, the newsgroups I used to participate in were full of dire warnings regarding HEAPX and Win 95, and I gave heed to them. But no, to be honest, no, I never did try it myself. So, if you want to give it a try, please do. BUT do make the *fullest* possible back-up before doing it. Care doesn't hurt. I believe it'll give you nothing more than a big intergalactic lock-up.:) But it might be worse. I'll fish up my old SYSTEM.INI and post the relevant parts tomorrow, but I think that what is in Q136776 should be enough, together with the HURRICAN.INI in the .7z I posted. As you use Win ME, I'm betting you'll be the first ever to try HEAPX on it...

Yes, Nuts & Bolts and NETROOM were Helix other products. I seems to me that it was Nuts & Bolts McAffe was really after. NETROOM was much better than either Qualitas 386-to-the-MAX or Quarterdeck QEMM, not to mention M$ EMM386, at that point in time...

All our problem does lie with the 16-bit heaps. The 32-bit heaps are large enough.
Alternatively, we could use more 32-bit applications.:)
The depletion of 16bit resources is created by the running of 32bit apps. The more you run 32bit apps the more the 16bit segments of resources are depleted.

If someone is able to explain what takes place and why, it would be great.

As far as I understand it (I could be wrong) user.exe 16 bit heap stores things like window classes, message queues and gdi.exe 16 bit heap stores logical pens, brushes so if any app 32bit or otherwise requires any of these, into the heap they go. Gdi.exe and user.exe heaps are in conventional memory the first 640kb of system memory and reside their for dos app compatibility a legacy from the first processors, which couldn't address over 1 MB of memory, in between is the uma for ROM, RAM on peripherals and memory-mapped input/output .

Gdi32.dll and user32.dll have no such limitations being purely for 32 bit apps, although the apps will still use some of the 16 bit heaps.

To overcome this everything in user.exe and gdi.exe has to reference a 32bit heap in the same manner as its 32bit counterparts, altering the global descriptor table to accomadate this, but this will break dos compatability. Whether as an alternative, the space of the uma can be absorbed as it is very rarely used these days, to increase the size of the 16bit heaps and keep dos compatability.

Just some thoughts.

oscardog is precisely right, AFAIK. HEAPX worked in Win 3.xx because it implemented a kind of "heap paging" to free up space in the USER and GDI heaps...

eidenk, here are the lines to add to system.ini:

====================================================

[386Enh]

Device=C:\HURRIC98\HEAPX.386

[Hurricane]

HurricaneDirectory=C:\HURRIC98 ; this is whatever dir you have put the files in.

WindowsDirectory=C:\WINDOWS ; this is %windir%

====================================================

As you can see I was wrong v 2.02 already was called Hurricane 98 (but still Helix Hurricane 98).

Hurricane 95 is v. 2.00 (the installer, HURRI.EXE, is dated jul 16, 1998).

Good luck! As soon as I can I'll test it too, in a sandboxed plain-vanilla Win 98SE.

But I think the system resources are simply too different between Win 3.xx and 9x/ME for it to work.

I've researched heap problems for some time, and i have some things to say:

> USER and GDI % you see in resource monitors are blatant lies; they are trimmed down to 99% after Explorer initializes, as Matt Pietrek says (in Undocumented Windows 95, if i remember correctly).

When you have 90% of USER and GDI resources, they're actually around 80% and 65%.

> GDI resources:

* I have "undocumented" structure references for Win3.1 floating around (for LOT of them!). They are mostly accurate for 9x too!

One more good thing: each GDI object has hOwner entry, which describes HTASK handle corresponding to task that allocated this GDI object. Bad thing: in 9x, a lot of entries may have bogus hOwner value. It may correspond to Win32 thread which died already, or was allocated by Win32 shared memory module, as it seems.

* The main local heap eaters are DCs. They're occupying ~266 bytes. That means that if your program allocates 30 device contexts, you can start only 7 copies of your program.

The good thing they're allocated as "moveable" structures, and memory DCs are usually not locked. Since that, "GDI Heap Extender" may be created, which will:

- Hook LocalAlloc/Lock/Unlock/Handle/Free. When moveable DC structure is allocated, it stores only some 'pointer' value, several bytes. When it is locked, it is actually allocated (moved back) into GDI local heap. When unlocked, it is moved back to our storage. I believe it is what Hurricane Heap Extender does.

-> I'm starting such a program. That won't be easy, since don't have any Win16 programming expirience. There are still a lot of questions, for example how would i allocate storage for real structures (if it'll be still 64K, there won't be much profit), will undocumented structures actually work, how dib and video drivers will react, etc.

>USER resources:

* That's completely another story. There are only fixed entries in local heap, and they're usually small. USER Local heap is exausted usually by thread message queries, or due to bug situations in Win32 when it becomes reentrant and loses track of itself.

As far as I understand it (I could be wrong) user.exe 16 bit heap stores things like window classes, message queues and gdi.exe 16 bit heap stores logical pens, brushes so if any app 32bit or otherwise requires any of these, into the heap they go. Gdi.exe and user.exe heaps are in conventional memory the first 640kb of system memory and reside their for dos app compatibility a legacy from the first processors, which couldn't address over 1 MB of memory.
I don't get it. Why are those things 16 bit for DOS compatibility? No DOS programs uses such things, as they're DOS programs, not Windows programs.
Enhanced mode

Enhanced mode Windows rode squarely on the two main features introduced with the 80386: paging and virtual 8086 mode. Paging offered Windows the ability to use vast amounts of memory, even if there wasn't a corresponding amount of physical memory. This was done by using disk space to simulate real physical memory. Virtual 8086 mode enabled Windows to run MS-DOS-based programs in a processor simulation of an 8086 CPU. Ill-behaved MS-DOS-based programs were rarely able to bring down the entire system anymore.

Interestingly, enhanced mode Windows was superior to standard mode primarily because of additions underneath the standard mode code. The layer of ring 0 code known as the Virtual Machine Manager (VMM), along with Virtual Device Drivers (VxDs), introduced the architecture that still powers Windows 98. This ring 0 code did many things, but the two most powerful were page-based memory management and virtual machines.

Enhanced mode Windows implemented page-based memory management, but this fact was essentially invisible to applications and the ring 3 system components; the VMM simply made a bigger pool of memory for the system to allocate from. All user mode code still dealt with segments and was oblivious to the fact that physical memory might not be assigned to a segment at any particular point in time.

Virtual machines allowed Windows-based applications and multiple MS-DOS prompts to run concurrently, with each believing that it had sole control of the keyboard, mouse, screen, and so forth. All Windows-based applications ran in a single virtual machine reserved for their use. This was made possible by the virtual 8086 mode support introduced with the 80386.

As an interesting note, the ring 3 components of enhanced mode Windows were nearly identical to the standard mode components. Components such as window management (USER.EXE) and graphics (GDI.EXE) used the same code in both modes. The only significant difference between the two modes was in the kernel (KRNL286.EXE versus KRNL386.EXE), and in how MS-DOS-based programs were run.

@Tihiy Great to hear, I wish you every sucess. I understand that the 2m heaps overlap the 128k heap (i am assuming this is the 2 x 64k heaps) and enables a single selector. I need to do some more research though to make things clearer

@dencorso thanks for the file, I have tested it under a minimised win98se system with a 95 shell and the bootlog shows the driver being successfully loaded. I created 2 system.ini`s 1 with the driver 1 without and loaded up on each gdi.exe with multiple winrars until system resources became critical, both seem to be able to handle 21 winrar windows open before gdi.exe reached 1%. I will try to run this on a 98 shell where I can monitor memory and see if the driver is actually doing its part. The system seemed fine with the driverinstalled and had no ill effects.

Edited by dencorso
Link to comment
Share on other sites

@BenoitRen: To complement what oscardog has just said, Windows Standard Mode was nothing more than a sophisticated DOS Extender...

@oscardog: Wow! You did it! :thumbup: Did you load both from system.ini, heapx.386 from a "device=" line in [386Enh] and heapx.drv from a "Heapx.Drv=<path>\HEAPX.DRV" in [boot]? Or how did you manage to have it load? Glad to know you didn't get just a systems-wide crash! This sure has to be explored further.

I have heapx.drv=c:\heapx\heapx.drv in [boot]

Device=C:\heapx\HEAPX.386 in [386Enh]

[Hurricane]

HurricaneDirectory=C:\heapx

WindowsDirectory=C:\MINDOWS

and altered Hurrican.ini to

[sysLoader]

GlobMem.Drv=C:\HURRICAN\GLOBMEM.DRV

Heapx.Drv=C:\heapx\HEAPX.DRV

WinGuard.Drv=C:\HURRICAN\WINGUARD.DRV

I am not sure if anything is being activated on the thresholds set as both with and without the driver free

resources seem to be the same despite resources being below them. Perhaps it has a dependancy on other

files, I will try to get the full package and see if that rectifys things. Bootlog shows the driver loading

successfully.

As a side note I see in the hurrican.ini references to cleanuponclose and compactonclose this reminded me of

the UserSeeUserDo function in user.exe which subfunctions allow allocate,free and compact in the user 16bit

heap. So at least this heap can be modified at will and I suspect this is what heapx is doing. Unfortunately as

Tihiy mentions this is the lesser of the problem heaps

Hi, oscardog!

I've done some testing myself, and I can cornfirm HEAPX.386 is loaded and initialized correctly, and then sits

there as a Stactic VxD without causing any ill effects to the system. All initialization steps are documented in

bootlog.txt, and both Norton Utilities System Information - Memory and APSoft VxDView find it in memory,

under the name VHEAPX (it has a PM API but offers no VxD Services). So far, so good. As it's harmless, it

may be of help to Tihiy's projected GDI Expander, provided we can understand what it does do. To me it

seems a monitoring VxD, but it also could be an interceptor VxD.

As for HEAPX.DRV, that's another story, entirely:

On one hand, the line heapx.drv=<path>\heapx.drv in [boot], in system.ini in totally ignored by Win 98SE,

during initialization. So, the system initializes normally, but there is no sign of heapx.drv anywhere in memory.

On the other hand, by adding it to the end of the devices= line in [boot], in system.ini, Win 98SE does try to

load it and soon gives a box that says "MSGSRV32 An error has occurred in your application. Etc." At this

point there are two choices: close or ignore. When I chose close, I then got "MSGSRV32 caused a GPS in

module HEAPX.DRV at 0001:00008BAE". I closed this message box too, and the system froze. Only

<ctrl><alt><del> still worked, but the only option I had was to shut down. When I choose ignore, I got an

instant BSOD, saying "Fatal Exception 0E has occurred at 01A7:BFF9E0B7". I told it to proceed, it returned to

the windows initialization and then gave me "MSGSRV32 caused a GPS in module KRNL386.EXE at

0002:00000102". I closed this message box too, and the system froze dead. Nothing worked anymore. After

restarting in DOS and editing out heapx.drv from the drivers= line, everyting returned to normal.

I'm sorry I wasn't able to bring good news, :( but those are my findings.

The cleanuponclose and compactonclose of the hurrican.ini was well worth the effort of examing heapx.

I have very little in the way of good gdi information, hopefully a book coming soon might help a little, from what

I have found is that the 32bit GDI heap start 128k past the 16bit dgroup and use the same selector and store

offsets relative to the base address of the gdi dgroup selector.

I was wondering that if this is the case, if the PENS,BRUSHES, etc could all be placed in the

32 bit heap. IsGDIObject figures out which heap it should look for the object in via 16 bit handles ending in

2,6,0xA or 0xE, handles for objects in the 32bit heap are multiples of 4. IsGDIObject then calculates this

address and constructs a pointer to it. I am not sure if HGDIOBJ would cause complications if this was to be

done, or if the header files would need modifying more.

I examined the code for GetObjectType which is the recommended alternative to IsGDIObject for 32 Bit Code.

It appears to process four types of Handles.

1. Selector Handles (value = 3 mod 4)

2. 32 Bit Mode Handles (value = 0 mod 4)

3. New Style 16 Bit Handles (value = 2 mod 4)

4. Old Style 16 Bit Handles (value = 2 mod 4)

The Selector Handles are not connected to the GDI Resource Heap so they are not relevant to the Resource

issue.

The 32 Bit Mode Handles are offsets into a 64K Index Table which contains 32 bit offsets to the 32 Bit

Resource Area. Offsets are relative to the 64K Resource Area.

The two styles of 16 Bit Handles are offsets into the 64K Resource Heap. They point to 4 Byte entries.

Bit 6 of the third Byte determines the style. If set, the entry is a New Style Entry, otherwise it is the Old Style.

In the New Style Entry, the first two bytes are a 32 Bit Mode Handle (See #2 above) which is used to access

the 32 Bit Resource Area.

In the Old Style Entry, the first two bytes are a 16 Bit offset within the 64K Resource Heap.

The Resource is laid out as follows:

64K Resource Heap

64K Index Table

32 Bit Resource Area

This explains the 128K separation between the two Resource Areas that Oscardog noted.

Considering how easy it is to set the New Style flag and use the 32 Bit Resource Area, I suspect there is a

reason why Microsoft did not move all Resources out of the 64K Heap.

In order not to repeat 13-year old research, here's the Windows 95 System programming secrets:

http://tihiy.ahanix.org/asmmetpitr.rar

Many thanks

@rloew I guess those remaining in the 64k heap are for backwards compatibility, whether users are willing to

forego this for larger system resources, or perhaps made selectable on boot

As Tihiy states and contrary to what we have been told before regarding resource limitations, the main

problem are DC`s. Particulary the over use in applications in using CS_OWNDC in creating private DC`s

which very quickly consume the heap, when apps i.e not involved in cad, video editing do not require a boost

in graphics performance and could well use common DC`s leaving much much more resources free.

The Resource code is mostly 16-Bit Code, thunked down from 32-Bits as needed. So far, it appears that each Object type is individually coded, so moving them would be a substantial undertaking to find all the references to each one.

Running 14 IE6 instances of NASA's "Today's Space Weather" used up all of the 16-Bit Resources. The most obvious object in the GDI 16-Bit Heap was the 40 Byte Bitmap Object.

Pardon me, but where did you find 40 byte bitmap entries?

Entry: Handle- 4210 Address- 51846 Size- 34 Flags- movble Locks- 0 Type - 5 Type - KO hOwner - 14598 Owner name - Iexplore
Entry: Handle- 2558 Address- 57062 Size- 46 Flags- movble Locks- 0 Type - 5 Type - KO hOwner - 10126 Owner name - Explorer

I've never seen 40 byte ones with various GDI versions.

Also, on topic: actually, my idea with hooking LocalAlloc/Lock etc is not able to solve anything, since (as

known from Matt Pietrek book) GDI does not uses LocalLock but just abuses locked handle as 16-bit pointer

into real local memory arena. :unsure:

I've never seen 40 byte ones with various GDI versions.
The total space used by each Bitmap record I saw was 40 Bytes. The record itself may be 34 Bytes with a 6 Byte Header.
Entry: Handle- 4210 Address- 51846 Size- 34 Flags- movble Locks- 0 Type- 5 Type - KO hOwner - 14598 Owner name - Iexplore
Entry: Handle- 2558 Address- 57062 Size- 46 Flags- movble Locks- 0 Type- 5 Type - KO hOwner - 10126 Owner name - Explorer

I assume the first entry in your sample is what I saw; they were IE Bitmaps after all. There could be other sizes, but I saw a lot of the first type.

The actual space used for these Bitmaps is 44 Bytes each if you include the Handle as well.

Note: There is no more. This is where the old tread broke off.

And now, here're some great news: :w00t:

The quotation below is post #522 from the UberSkin thread:

Tihiy, have you tested (extensively) UberSkin under Win98SE with the patched GDI32.DLL/GDI.EXE 4.90.3003 ? I can't think of a better reason for the (occasional with UberSkin 8.3.6 and much more frequent with later versions) out-of-the-blue GDI crashes (while resources are at 40-50% or better).
Yep, i figured out GDI troubles. Those crashes / hangs / BSODs are caused by bugs in programs you're using (Miranda as one of them). I've created technology called GDI salvation which successfully fights with buggy programs and resource leaks. Stay tuned.
Edited by dencorso
Link to comment
Share on other sites

@eidenk: You're welcome! I'm glad you started a topic on this subject once more.

@blackwire: Well, Microsoft said it, about Hurricane and Win 9x. And also said: "Upgrade to Helix Hurricane version 2.0 or later." All along, Helix also said that no version of Heap Expander was compatible with Win 95. Note that, for Win 3.1 and WfW 3.11, Heap Expander worked beautifully, all right! But when Microsoft modified the workings of the resources, with Win 95, Helix was at the end of its run, and never developed the Win 95 version. :( In our tests oscardog and I used the latest available version of Heap Expander, just to see what happened, and to perhaps get some ideas out of it. It seems we got nowhere, after all... But that's past already.

For the present, I prefer to believe Tihiy's GDI Salvation will solve once and for all the GDI resources problems, that's why I'm so excited about it. :thumbup Then, half of the problem will have been solved, there remaining just the USER resources problem. I believe Tihiy said somewhere, around late 2007 or early 2008, that he thought his own approach wouldn't work also for the USER resources.

And then he went silent about the resources problem, AFAIK, until that post I quoted at the end of post #4...

Edited by dencorso
Link to comment
Share on other sites

Tihiy's GDI Salvation will solve once and for all the GDI resources problems, that's why I'm so excited about it.
Unfortunately NO. As i said, it fights with buggy programs and resource leaks. Not more. It can't expand heap beyond 64K.
Link to comment
Share on other sites

Tihiy's GDI Salvation will solve once and for all the GDI resources problems, that's why I'm so excited about it.
Unfortunately NO. As i said, it fights with buggy programs and resource leaks. Not more. It can't expand heap beyond 64K.

OK. Right. Maybe I'm overenthusiastic. Nothing can really expand 16-bit segments beyond 64 KiB, of course. That's built-in on the design of the 80X86 family. But recovering from buggy programs and resource leaks is a giant step forward. Well-behaved programs rarely exhaust the GDI resources. I was thinking somewhat along the line of recovering from unexpected behaviour when I dreamed about the putative "sfix.vxd" (at the 1st quote at the top of post #3). rloew didn't find the ideia workable, but, then again, maybe he was thinking on really solving all types of resource exhaustion. My ideia was, and remains, that even solving just some of them is worthwhile. Now, please, do tell me: will GDI Salvation have a stand-alone version, or do you intend to release it only integrated with UberSkin? I'm sure that a stand-alone version will be much welcome, and reach a broader user base. I think UberSkin rocks. But I do envisage it as a separate entity from programs dealing with the resources issue. And thanks a lot for returning to this discussion, now that it has been restarted. You do rock! :thumbup

Link to comment
Share on other sites

Now, please, do tell me: will GDI Salvation have a stand-alone version, or do you intend to release it only integrated with UberSkin?

Have you looked at/used UberSkin? You don't have to use any of the skinning to get its other benefits (which right now are primarily taskbar locking, closing hung programs by right clicking them on the taskbar, and resolving file copying hanging Explorer with IE6). A release of UberSkin that doesn't include any skins and has a different name, like UberHeap (kidding), could be an option but totally isn't necessary. Just make it clear that UberSkin is awesome (and fixes some Windows bugs) and that using the skins isn't required, and that it's worth installing on any 98SE/ME machine.

Queue

Edited by Queue
Link to comment
Share on other sites

Since i fail to deliver guide, i'll tell you how GDI Salvation works here:

1) Handle protection. 9x lacks proper GDI handle validation which may result in incorrect memory access or wrong resource deletion and as result crash, hang or BSOD. Which is not-so-rare: i know a lot of programs which try to delete deleted handles or wrong handles (icon handles for example).

2) Handle arena cleaning. GDI relies on moveable 16-bit handles. Too bad that they require significant memory overhead, since handle arenas are allocated in heap too. Worse, they're never freed (by design?): that's why when you quit heavyweight programs GDI resources don't go up back. Even worse, they can fill most heap and make whole system slow crashing a**. RP9 addresses this by running through those arenas and removing free ones. This run is performed after every app termination or when resource allocation fails. It's disabled for first 60 seconds of system uptime and you can diable it through RPConfig.

Test numbers: try running programs which consume much GDI. I've runned 3 ImgBurns. After exiting them, without GDISalv 9% of GDI were lost.

3) Leak protection. Bizzare bug, but 'extended pen' GDI objects are somehow not marked with owner, meaning they can leak permanently. This was corrected in 9.0.2.

USER Salvation was explained before. In short, it saves >100 bytes per process by combining comctl32 classes with system classes. Not much, but you can run 60% more Notepad copies on clean system.

Writing a proper heap expander, with proper performance and compatibility is an extremely hard task.

Link to comment
Share on other sites

Very interesting details! Can't wait to test this technique (be it in alpha/beta stage - I'm used to that).

I suspect my sudden GDI crashes may occur due to overloaded registry (16MB+ all in all) and I'm extremely curious if this technique can help in this case.

Writing a proper heap expander, with proper performance and compatibility is an extremely hard task.
And who else would be the best one to take a stab at it...? :rolleyes:
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...