32bit Windows not using/seeing all 4GB RAM?

[cluberti]

This question is theoretically for educationally use ONLY.

The 4GB RAM limit of Windows 7 (as in vista) is known to be software limited from MS. However there is a "workaround" using certain legit(!) files from Windows Server 2008 if you own both versions, replaced with Vista system files for Windows Vista to take advantage of more than 4GB of RAM. As Server 2008 can handle up to 64GB RAM.

Is this possible also in the Windows 7 kernel?

This is a good post for a sticky, so I'm going to re-use this thread for educational use and strip out the name of the person asking the question:

First, we'll discuss the 4GB RAM limit on x86 systems. The 4GB limit is hardware-based (well, almost - I'll explain the workaround later that allows up to 128GB) - the x86 architecture can really only address memory addresses at 2^32, or 4,294,967,296 (4,294,967,296/1024x1024=4096, or 4GB) at any one time. So, no executable code can be at any address higher than the 4GB "window" that the x86 CPU can see. However, there were 2 workarounds that Intel put in place awhile ago to try and extend the life of the 32bit platform until it could get it's 64bit Itanium platform mainstream, and these were PAE and PSE (read here for more on why Microsoft chose to implement PAE rather than PSE).

The specific workaround used in Windows is called Physical Address Extension (PAE), which was added by Intel in the Intel Pentium Pro and Pentium II lines of CPUs (and continued subsequently to this day) to allow data (and data ONLY) to be mapped into RAM above the 4GB memory address range - however, there is a set of limitations on this. The first limitation is that this window to the RAM above 4GB *must* be mapped into a contiguous memory address range in the application's original 2GB (or 3GB, with /3GB) process space, because again the x86 architecture can only address memory up to the 4GB range (so mapping it into this window allows it to be addressed, with the application handling the management overhead). That brings us to the second limitation, specifically that this window must be managed entirely by the app, as I mentioned previously - there is no using the OS's memory manager, as the memory manager does not know how to handle memory above the 4GB boundary in a process' space, thus placing an overhead on the app. The last limitation is that the memory window *must* not contain executable code (again, the CPU's eip register doesn't understand addresses above 0xFFFFFFFF, the address limit of the x86 architecture). PAE is technically a hack, and if you want your applications to *really* use memory above 4GB, you will want to use a CPU architecture that supports it (Intel's Itanium ia64 or the AMD/Intel x86-64, or x64).

Note that there are ways to get a 32bit system to use and run applications above the normal 4GB boundary, as Windows Server editions can do this natively when /PAE is enabled. Microsoft has limited this (artificially) on client systems, however, due to potential for client-side driver incompatibility, as well as the fact that x86-based servers with more than 4GB of RAM have been fairly common for awhile, whereas 4GB client systems were fairly rare and probably not as well tested with all of the possible myriad of drivers that can be found on a client system. Addressing that much memory can also increase the size of the PFN entries, doubles the size of each PTE, and otherwise starts to constrain the kernel's normal 2GB of VA. For example, an x86 Windows system with 16GB of RAM is already using ~112MB of VA just for PFN entries, and you take the available PTEs available to the system and reduce them by approximately 7x.

While I don't agree (at all) with doing this as it violates the EULA and license you have to run a Windows client OS, as well as cripples the OS kernel in many ways, Geoff Chappell has an article about how the 4GB restriction on client OSes can be bypassed here. If you want to use more than 4GB of RAM with an actual legally licensed copy of Windows client, however, you need an x64 license.

The section portion of this sticky is the problem people see when they boot up an x86 system with 4GB of RAM, but Windows only reports the available RAM as somewhere between 3GB and 4GB. Rather than be lengthy about it and rehash old threads, I'll simply point to them to explain why this happens:

1. Start here, here, and here for a technical overview of how this can happen, and

2. Read this, this, this, and this to see real-life examples of this happening, and what can (and cannot) be done to resolve it.

Now it's been stickied, so if you ask these types of questions going forward, you will simply be pointed to the contents of this thread. These are not necessarily Windows problems, these designs and limitations are an outcome of the design and architectural limitations of the 32bit architecture - especially the second issue, where the BIOS reserves memory for hot-add/hot-swap memory, PCI-X/PCI-E buses, and (if it applies) your 256MB or 512MB video card reserve memory resources for their usage, causing the discrepancy in reporting of installed memory vs *available* memory to the OS.

Last updated 2 August 2010