Jump to content

Take advatage of 4GB+ RAM in Windows 7 32-bit


BoardBabe
 Share

Recommended Posts

...plus the limitation of not being able to execute code there (even the driver would have to move the code down into a window inside the 4GB boundary to execute it like an app would).

This has me a little confused, if the RAM above 4GB is mapped as a RAMDISK, and the swapfile is located there, how would it perform any differently than a swapfile on a hard drive (other than being much faster since, well, RAM is fast)? What I mean is, I don't understand your comment about needing to ''move the code.'' I do understand that, using PAE, code above the 4GB barrier can't be directly executed, but that's not what would be going on in this case; the RAM above 4GB is just a RAMDISK used to hold a swapfile.

CoffeeFiend, your explanation was exactly what I meant by ''temporary storage,'' but I appreciate the explanation. The nit-picking over me saying 4GB (as opposed to 2GB being usable by a single process) kind've led you off track; the concepts were the important part, namely virtual memory letting programs share the RAM. I get it, one program can only use 2GB under normal circumstances. This is primarily an academic discussion; I have nothing against using a 64-bit OS, but being able to get more out of the hardware with a 32-bit OS is the topic at hand.

Queue

Edited by Queue
Link to comment
Share on other sites


...plus the limitation of not being able to execute code there (even the driver would have to move the code down into a window inside the 4GB boundary to execute it like an app would).

This has me a little confused, if the RAM above 4GB is mapped as a RAMDISK, and the swapfile is located there, how would it perform any differently than a swapfile on a hard drive (other than being much faster since, well, RAM is fast)? What I mean is, I don't understand your comment about needing to ''move the code.'' I do understand that, using PAE, code above the 4GB barrier can't be directly executed, but that's not what would be going on in this case; the RAM above 4GB is just a RAMDISK used to hold a swapfile.

Queue

The driver still needs to map the addresses of memory from their *real* locations into a "window" in it's 2GB of kernel VA (drivers loaded in kernel load in the kernel's VA, and still need to use the AWE APIs to do this). You don't get a free lunch just because you're a driver or a RAMDISK. It's likely the RAMDISK is not going to suffer a noticeable perf hit when using / accessing the swapfile via the driver because it would be a kernel-mode filter driver, hence mapped into nonpaged kernel pool, and thus everything it does is going to be done from RAM ;). In the 32bit (x86) world, even as a driver, you still have to use the AWE APIs to map and manage those RAM addresses - the memory manager for the OS won't help you.

I really cannot explain this at a lower level, as you're getting into core memory manager code, and architectural limitations on the x86 instruction set. If you want to know more, I highly recommend Windows Internals 4th Edition from Mark Russinovich, specifically chapter 7.

Link to comment
Share on other sites

  • 2 weeks later...

Yes, definitly leave your 'explainations' (ie biased opinions) to correct documentation, like Russinovich sure, or even more obvious, wikipedia. I think your sticky about this should simply have linked to such sources.

Actually, I don't get why people first ask some well meaning hack these kinds of questions when the internet lets you get it straight from the experts instantly -- no waiting for a reply, elegantly explained, and largely unbiased.

To each their own, I guess.

I think the industry (chip and os architects) screwed up and now everyone is left with two less-than-ideal options: 32bit/PAE or AMD64. Ideal would be a design that doesn't force the native integer size (data) to have the same number of bits as an address. And why, to maintain pre-ANSI C compatability? Honestly, compared with all the other marvelously intricate capabilities built into moden processors, compilers, and operating systems, this obvious separation would have been trivial to achieve. What are we going to have next, 128 or 256 bit addresses further bloating our code with useless zeros because the processor just happens to have 128 or 256 bit general-purpose registers inside and a few apps would like to take advantage of them for very-long-integer calculations? *puke*

And no-one tried to answer the original question, which I think is of interest -- I too would like a way to get around the Microsoft-jerkoff-disabling of access to all of my motherboard ram unless I give them $1000's for (essentially) the same 32bit OS but with the word 'Server' in the name.

Link to comment
Share on other sites

Yes, definitly leave your 'explainations' (ie biased opinions) to correct documentation

followed shortly by biased opinions, nice.

and you would also have to add the words 'datacenter' 'advanced' or 'enterprise' as 'standard server' does not acheive anything over the 4GB mark.

Also a good article on PAE and /3GB switch.

http://russkaufmann.spaces.live.com/Blog/c...g!229.entry

Edited by iamtheky
Link to comment
Share on other sites

Yes, definitly leave your 'explainations' (ie biased opinions) to correct documentation, like Russinovich sure, or even more obvious, wikipedia. I think your sticky about this should simply have linked to such sources.
Correct, because posting an "in layman's terms explanation" for the vast majority of the people who can't figure this out in technical terms for themselves is somehow creating bias. Great, you know math and probably some electrical engineering, and understand the concepts - guess what, the vast majority of people in the world do not. And you should probably re-read the definition of the word "bias", because there's none in my post - if you think "bias" means "reiterating facts", then yes, I'm biased.
I think the industry (chip and os architects) screwed up and now everyone is left with two less-than-ideal options: 32bit/PAE or AMD64. Ideal would be a design that doesn't force the native integer size (data) to have the same number of bits as an address. And why, to maintain pre-ANSI C compatability? Honestly, compared with all the other marvelously intricate capabilities built into moden processors, compilers, and operating systems, this obvious separation would have been trivial to achieve. What are we going to have next, 128 or 256 bit addresses further bloating our code with useless zeros because the processor just happens to have 128 or 256 bit general-purpose registers inside and a few apps would like to take advantage of them for very-long-integer calculations? *puke*
So I'm sure the 8088 was good enough for you, because programs were designed that didn't need to take advantage of the additional 16 registers in a 32bit processor? There *are* applications that run well on 64bit, and if everyone thought like you, we'd all be running either a 16bit 8086 or a true 64bit only like the Itanic. Yes, let's hold back progress out of the limited 32bit architecture just because most apps won't use it yet. Let's ignore the fact that there's really no problems with moving to an x86-64 architecture on 64bit hardware, and throw out the "why if only a few apps will actually use it!" argument. Because holding up progress that really doesn't bring with it *any* perceptible problems to the architectural design of the system, and yet allows for it to be able to do far more, both now and into the future, is definitely a great idea.

And as to the design of the x86 architecture, no one screwed you. It was designed when even 1MB was a *huge* amount of memory, let alone 16MB or 160MB and it was the intent that we would be using an x64 architecture before we hit the 4GB limit of the CPU's design, or so they thought. In fact, Intel *failed* with the original Itanium to gain any traction anywhere but the high-end database server market, and AMD's x86-64 was seen (and ultimately won as the choice for x64 on the desktop and server platforms) as a far better design for both the x64 platform in general, and also the migration of x86 to x64 (because it could run x86 code natively on the CPU without using an emulator, like the Itanic does). Again, you're free to your opinions, but the rest of us are free to look at them and think they're crazy.

And no-one tried to answer the original question, which I think is of interest -- I too would like a way to get around the Microsoft-jerkoff-disabling of access to all of my motherboard ram unless I give them $1000's for (essentially) the same 32bit OS but with the word 'Server' in the name.
The original question was answered - it's not the OS (or Microsoft "jerking you off" or whatever you think). It's a limitation of the design of the 32bit architecture, and if you don't get that then no amount of explaining this is going to get you to understand that, I guess. Adding PAE to a system via Windows' AWE API set is something that you can "get around" by simply purchasing a 64bit version of the OS, for the same price as the 32bit version. Same price.
Link to comment
Share on other sites

Yes, definitly leave your 'explainations' (ie biased opinions) to correct documentation, like Russinovich sure, or even more obvious, wikipedia. I think your sticky about this should simply have linked to such sources.

There is no bias in that, it's strictly factual. And stating wikipedia as the "obvious correct documentation" is kind of funny, as more or less any hack could have written that page (even heavily biased hacks) -- over references like Windows Internals 4th Ed no less.

I think the industry (chip and os architects) screwed up

So what you're saying is: you know better than all of the most knowledgeable industry experts, both on the hardware and software side (or what they should have done at least), who all seemingly screwed up? :rolleyes:

now everyone is left with two less-than-ideal options: 32bit/PAE or AMD64

Perfect solutions to any problem hardly ever exist. "Good enough" (i.e. compromises) is what we usually end up using. PAE is barely what I'd call a solution in itself (more like an ugly hack around a real limitation). AMD64 actually solves the problem quite nicely. We do get a flat memory model with access to > 4GB (no fancy tricks to manipulate more address lines on the CPU than EIP has bits which greatly complicates things) and long mode to go with it. We get a larger physical and virtual address space. We finally get more [general purpose & SIMD] registers and -- bigger ones indeed (they're actually useful too). And we move away from like a half-dozen calling conventions (cdecl, stdcall, etc) down to one single, clean, modern convention. It's a great move overall.

I too would like a way to get around the Microsoft-jerkoff-disabling of access to all of my motherboard ram unless I give them $1000's for (essentially) the same 32bit OS but with the word 'Server' in the name.

Except it's not actually the same OS. And only allowing PAE in server-class OS'es back then made sense, as it was pointless on a desktop anyways, since you need *all* your software to be written specifically to use it. And basically, only server software made use of it -- very little of it actually: the only one I've encountered so far is SQL Server. Not that it actually solves this problem either, as you can't run code in there anyways (only useful in cases where you have many GBs of data to load), and it doesn't fix any other issues with the old 32 bit memory model (adress space and other limits). PAE isn't a "general purpose" way of going past 4GB. And you don't need to pay 1000's of $ either as cluberti said.

don't feed the troll guys

Indeed, I shouldn't have :blushing:

Link to comment
Share on other sites

Reading plainly what BoardBabe has actually written, I contend that the limitation to which he is refering is this one:

Physical Memory Limits for Windows Releases

and not some other random limitation like the largest pickel that he can swallow or the 4GB virtual memory limit in processes for all editions of 32-bit windows or the reading comprehention of cluberti.

So, for example, suppose I have a motherboard with 16GB of physical ram installed. With Vista 32-bit, if you add the total physical ram allocated to each of the running processes, it will never total more than 4GB, even though PAE is enabled. On the other hand, Server 2003 enterprise will, for example, allocate 3GB of the RAM to one process and a totally different 3GB to another process, and so on with additional processes until all 16GB of motherboard ram is in use.

BoardBabe claims that this limitation can be gotten around in the case of Vista 32-bit by grabbing some files out of Server 2008 (after cautioning its legality):

The 4GB RAM limit of Windows 7 (as in vista) is known to be software limited from MS. However there is a "workaround" using certain legit(!) files from Windows Server 2008 if you own both versions, replaced with Vista system files for Windows Vista to take advantage of more than 4GB of RAM. As Server 2008 can handle up to 64GB RAM.

Finally he asks:

Is this possible also in the Windows 7 kernel?

And as I said, noone in this thread has addressed either his claim or his question.

Edited by seedofonan
Link to comment
Share on other sites

I contend that the limitation to which he is refering is this one:

Memory Limits for Windows Releases

and not some other random limitation like the largest pickel that he can swallow

Which are the very same limitations we've been discussing from the very start. Yeah, others need to work on their reading comprehension...

BoardBabe claims that this limitation can be gotten around in the case of Vista 32-bit by grabbing some files out of Server 2008

Which like we said from the start is mostly inaccurate. You can enable PAE, but the ONLY thing that changes is that special apps such as SQL Server could use the extra RAM, and for data ONLY. It DOESN'T make your everyday apps make use of 4GB+ (either per process, or for for all processes combined), and as such it's not just not a solution to the problem (besides not solving other issues and limitations), just as cluberti very well explained in post #3

And as I said, noone in this thread has addressed either his claim or his question.

Except, we already said x64 versions (Win 7 included) already do support PAE (not that you need it in the first place). As for 32 bit version, again, it doesn't solve the problem anyways, and using chunks from an older kernel into Win 7 is unlikely to work (Vista and Win 2008 use the same codebase, so it's not surprising bits and pieces can be mixed).

Link to comment
Share on other sites

BoardBabe claims that this limitation can be gotten around in the case of Vista 32-bit by grabbing some files out of Server 2008

Which like we said from the start is mostly inaccurate. You can enable PAE, but the ONLY thing that changes is that special apps such as SQL Server could use the extra RAM, and for data only. It DOESN'T make your everyday apps make use of 4GB+, and as such it's not just not a solution to the problem (besides not solving other issues and limitations).

BoardBabe never said anything about what 'problem' she may or may not have been interested in solving. What you say is inaccurate -- that enabling PAE in 32-bit Vista (it is by default) will let special apps such as SQL Server use the extra RAM. In fact the Vista 32-bit kernel (software) prevents that. And beyond that failing, neither can two separate apps both allocate 3GB a piece. For either of those scenarios to work, one would have to pay Microsoft $1000's of dollars for a high-end 32-bit Sever edition of Windows, because they do not include the software limitation in the kernel to which BoardBabe is refering.

I contend that the limitation to which he is refering is this one:

Memory Limits for Windows Releases

and not some other random limitation like the largest pickel that he can swallow

Which are the very same limitations we've been discussing from the very start. Yeah, others need to work on their reading comprehension...

lol, now you're among the 'others'.

And as I said, noone in this thread has addressed either his claim or his question.

and using chunks from an older kernel into Win 7 is unlikely to work (Vista and Win 2008 use the same codebase, so it's not surprising bits and pieces can be mixed).

Obviously. But the original question remains -- will an analagous mixing of same-codebase kernels between high-end server and desktop versions of Windows 7 still get around this software limitation like BoardBabe claims it does with Vista/Server08, or did Microsoft put the kibosh on that?

Link to comment
Share on other sites

BoardBabe never said anything about what 'problem' she may or may not have been interested in solving.

I think it was fairly obvious to most people.

What you say is inaccurate

Only because you misread it. And yes, again you blame others...

Keep trolling... I'm done wasting time answering anything you write. I should've listened to bledd obviously.

Link to comment
Share on other sites

So, for example, suppose I have a motherboard with 16GB of physical ram installed. With Vista 32-bit, if you add the total physical ram allocated to each of the running processes, it will never total more than 4GB, even though PAE is enabled. On the other hand, Server 2003 enterprise will, for example, allocate 3GB of the RAM to one process and a totally different 3GB to another process, and so on with additional processes until all 16GB of motherboard ram is in use.
Would you care to quote your source on this? Because it is totally, utterly, wrong. A running Windows process knows *nothing* about the RAM underneath, only the memory manager does. And the memory manager can only map VA to either physical RAM or to paging file space that it can address - in 32bit, it is *4* gigabytes of RAM, and if you enable the PAE kernel, 32 to 128GB of paging file depending on the version of Server, and *no more*. If a process wishes to make use of actual, *physical* memory above the 4GB boundary on a 32bit system, it *has* to do so itself, *without* the help of the NT memory manager, using the PAE kernel and the AWE API set as I've mentioned above, and it *must* incur the overhead of mapping the memory addresses into a VA location within it's 2GB or 3GB (depending on whether or not /3GB is enabled) of VA so that it can be addressed when necessary, and the app must do the translation and memory managemtn.

If you're running a Windows service (like Terminal Services, for instance) that is written to take advantage of the PAE kernel, then yes, you can see Windows Server editions actually using more than 4GB RAM - but again, the limitation of *only storing data*, and *no executable code* persists even to Windows, and it's important to denote that it's not technically any Windows binaries using the additional memory, it's Terminal Services (in this example). It is not physically possible to execute code in an address space above the 4GB boundary, as FFFFFFFF is the largest area that can be addressed by a processor in 32bit mode, period. Anything higher will cause an exception.

And as I said, noone in this thread has addressed either his claim or his question.
Yes, it has been. Just because you do not get it doesn't mean the answer isn't here. It's been discussed ad nauseum here and elsewhere, and unfortunately you fail to grasp memory management in Windows - I'm sorry, but you do. Please, please read Windows Internals, either the currently shipping 4th edition or grab the 5th Edition when it releases later this year before posting about this again here. If you want some more information, please let me know, but what you currently understand seems to be incorrect.
Link to comment
Share on other sites

And just to put the final nail in the PAE Coffin.

HW driver must be able to handle PAE systems to work correctly in that enviroment. Since most XP drivers released by third party vendors doesn't PAE on Client Operating Systemes tends to be a bluescreen generating event. For 64 bit systems this problem doesn't exist either

Link to comment
Share on other sites

After discussion of this offline with seedofonan, I finally get what is being asked, so I will publicly acknowledge my misunderstanding of the question and offer my apologies to all involved for the misunderstanding. He's talking about Windows taking an application's 4GB of shared kernel/process VA, and having the Windows memory manager physically placing it above 4GB on the system (say, physically locating mspaint.exe's VA in pages above 4096), and on an x86 server OS that supports this via /PAE, yes this is possible. Henceforth, a public mea culpa for not understanding the question clear enough - and if Boardbabe was asking this, then I apologize as well because it was a misunderstanding of the question on my part.

x86 Windows client OSes are indeed limited artificially to 4GB and no more, due to the 64bit address space shown to drivers and potential for issues (real or perceived - I'm not aware of any drivers that would crash when confronted with 64bit addresses vs 32bit addresses on load in the last few years, but I guess it's possible). There's also the memory manager overhead, as the server OS has to translate the memory address above the 4GB boundary for either the kernel or process' share of the VA down to a 32bit range, although on newer machines I do not know what kind of a performance hit this would incur.

I also admit I hate it when I'm wrong in understanding a question. My apologies to all involved, it's my misunderstanding. However, I do 100% stand by my statement that there is no real detriment to switching to a 64bit OS (and skipping the PAE hack and performance overhead you will incur), and that for an app to directly access RAM above 4GB on Windows it needs AWE (Windows itself is using a form of AWE to perform the hack), but I will admit publicly that I misunderstood the question. What I posted is technically correct info, but not in answer to this question. And to answer the original question, I guess it would be possible to hack the 2008 R2 memory manager down onto Win7, although it'd probably be a violation of the EULA.

Link to comment
Share on other sites

  • 2 weeks later...

I remember Mark Russinovich specifically blaming video card drivers causing crashes under XP with more than 4GB.

[EDIT] Found it: http://blogs.technet.com/markrussinovich/a...21/3092070.aspx

However, by the time Windows XP SP2 was under development, client systems with more than 4GB were foreseeable, so the Windows team started broadly testing Windows XP on systems with more than 4GB of memory. Windows XP SP2 also enabled Physical Address Extensions (PAE) support by default on hardware that implements no-execute memory because its required for Data Execution Prevention (DEP), but that also enables support for more than 4GB of memory.

What they found was that many of the systems would crash, hang, or become unbootable because some device drivers, commonly those for video and audio devices that are found typically on clients but not servers, were not programmed to expect physical addresses larger than 4GB. As a result, the drivers truncated such addresses, resulting in memory corruptions and corruption side effects. Server systems commonly have more generic devices and with simpler and more stable drivers, and therefore hadn't generally surfaced these problems. The problematic client driver ecosystem lead to the decision for client SKUs to ignore physical memory that resides above 4GB, even though they can theoretically address it.

Related:

http://support.microsoft.com/kb/929605

http://msdn.microsoft.com/en-us/library/aa366778.aspx

http://support.microsoft.com/kb/294418

Coincidentally Sysinternals recently released a new program called Vmmap which is a process virtual and physical memory analysis utility.

http://technet.microsoft.com/en-us/sysinte...s/dd535533.aspx

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.


×
×
  • Create New...