Jump to content

Take advatage of 4GB+ RAM in Windows 7 32-bit


BoardBabe

Recommended Posts

  • 2 weeks later...

(and skipping the PAE hack and performance overhead you will incur)

and that for an app to directly access RAM above 4GB on Windows it needs AWE

There are some rather big misunderstandings about how this stuff works. Just to take the two above.

1) XP/SP2 and Vista runs in PAE mode by default (the DEP thinggy). So the overhead is there anyways. In PAE mode you have a 3 layer page table model. In 64bit mode it is not 3, but 4 layers - just to compare.

2) Applications/software/machine code instructions only sees virtual memory addresses. Virtual addresses are mapped onto the physical addresses via the page table (managed by the OS).

AWE is used if an application needs to use more physical ram than can be mapped into its virtual address space at one time (it will then slide the AWE-window around). It is not a way to address RAM above 4G address line. AWE is NOT PAE. You don't even have to run in PAE mode to make use of AWE.

A process' virtual address space can point to any place in the physical address space, below, above whatever. It doesn't matter one bit to the application since it only operates with virtual addresses.

The first limitation is that this window to the RAM above 4GB *must* be mapped into a contiguous memory address range

as the OS does not know how to handle memory above the 4GB boundary

The last limitation is that the memory window *must* not contain executable code

It doesn't have to be contiguous. There is no difference in ram below and ram above. Under normal operation, the page size is 4kB. So 4kB is the only contiguous ram you need to have.

The OS does know about RAM >4G.

Executable code can be wherever it sees fit, because of the virtual address space thing.

Edited by ksoren
Link to comment
Share on other sites

1) XP/SP2 and Vista runs in PAE mode by default (the DEP thinggy). So the overhead is there anyways. In PAE mode you have a 3 layer page table model. In 64bit mode it is not 3, but 4 layers - just to compare.
Correct, otherwise DEP wouldn't work. I'm well aware of it.
2) Applications/software/machine code instructions only sees virtual memory addresses. Virtual addresses are mapped onto the physical addresses via the page table (managed by the OS).
Correct, processes on a Windows machine only know about VA, not physical addresses, but the memory manager does the translation, and as such it does understand memory addresses AND VA.
AWE is used if an application needs to use more physical ram than can be mapped into its virtual address space at one time (it will then slide the AWE-window around). It is not a way to address RAM above 4G address line. AWE is NOT PAE. You don't even have to run in PAE mode to make use of AWE.
AWE is the API set used to access PAE memory. And technically, you are correct, you don't need /PAE to use AWE, but without /PAE (or running an OS that doesn't automatically enable PAE by default), you won't load the PAE kernel without /PAE. Thus your AWE calls *won't* be allocating memory above 4GB (it's just an API - it doesn't *require* PAE to work, but it *does* require PAE to access memory above 4GB, requiring the PAE kernel to do so, thus *requiring* PAE if you want it to do what it was actually designed to do:
Without PAE, AWE cannot reserve memory in excess of 4 GB.
A process' virtual address space can point to any place in the physical address space, below, above whatever. It doesn't matter one bit to the application since it only operates with virtual addresses.
True, but if you want that physical memory location to be *above* 4GB, you need the OS to load the PAE kernel, hence the need for /PAE. Also, due to limitations on the client and lower-end server OSes, you also need that OS to be an Enterprise or higher version of a Windows Server OS as well.
Executable code can be wherever it sees fit, because of the virtual address space thing.
Really? Because Microsoft says otherwise:
No Executables or Buffers within an AWE Window

The AWE API does not permit executable code (.exe, .dll, .sys, and so on) to execute from within an AWE window region in the process’s virtual memory or in the mapped physical memory pool the process is utilizing.

A similar restriction is that AWE window address ranges and memory pools cannot be used as data buffers for graphics or video calls.

Note Applications using the AWE API set must have the Lock Pages in Memory privilege in order to allocate physical memory. For samples and more information, see the AWE API information in the Microsoft Platform SDK.

Finally, it is not possible to free only part of the AWE memory allocation made by a thread or process. Each AWE memory allocation must be freed as a unit.

Link to comment
Share on other sites

but the memory manager does the translation

It manages the page table. The translation is done by the CPU.

AWE is the API set used to access PAE memory.

No. Technically, AWE has nothing to do with PAE. It is just a technique for a 32bit app to access very large armounts of RAM, but it is not PAE.

I also don't like the name "PAE memory" very much. It is just one big chunk of memory.

Really? "No Executables or Buffers within an AWE Window"

There was some confusion about a process' virtual address space couldn't point to to physical ram above 4G. This was my point. If the VA points >4G then you have executable code above 4G (you are not using AWE here).

Edited by ksoren
Link to comment
Share on other sites

AWE is the API set used to access PAE memory.

No. Technically, AWE has nothing to do with PAE. It is just a technique for a 32bit app to access very large armounts of RAM, but it is not PAE.

Again, you're technically accurate, but missing the point. Yes, AWE is just a way to use VirtualAlloc to do your own memory management. However, without the PAE kernel loaded, the AWE api's cannot access memory over the 4GB address limit. They'll work, mind you, but the ranges they get back from VirtualAlloc will be within the 4GB range. Edited by cluberti
Spelling
Link to comment
Share on other sites

but the memory manager does the translation

It manages the page table. The translation is done by the CPU.

I think you're the one who's not processing properly. Who provides the code to the CPU to do the conversion? Oh, yea, the memory manager.

AWE is the API set used to access PAE memory.

No. Technically, AWE has nothing to do with PAE. It is just a technique for a 32bit app to access very large armounts of RAM, but it is not PAE.

If you have a bicycle without handle bars, do the handle bars have nothing do with the bicycle and therefore the handle bars are just a way to drive the bicycle?

I also don't like the name "PAE memory" very much. It is just one big chunk of memory.

I really doesn't matter that you like it very much, a little or at all. If you need to simplify the terminology to your understanding, so be it, because you obviously have no real understanding of how any of these systems work.

Really? "No Executables or Buffers within an AWE Window"

There was some confusion about a process' virtual address space couldn't point to to physical ram above 4G. This was my point. If the VA points >4G then you have executable code above 4G (you are not using AWE here).

Considering that you cannot point to a Virtual Address above 4GB using a 32 bit application without AWE... perhaps you've discovered a way to create a Virtual Address wormhole using your flux capacitor.

Edited by jcarle
Link to comment
Share on other sites

Forget about AWE for a minute, okay? This has nothing to do with PAE. PAE just extends the PTE from 32 to 64 bits.

Considering that you cannot point to a Virtual Address above 4GB using a 32 bit application without AWE

Virtual address > physical address is done through the page table and has nothing to do with AWE.

Who provides the code to the CPU to do the conversion

Yes, the OS manages the page table. But the actual translation when a machine code instruction requests data on a memory address is done by the CPU.

10 processes (with a 2GB user space) can potentielle make use of 20 GB RAM without ever touching AWE. Each process just points to different places in physical ram.

because you obviously have no real understanding of how any of these systems work

Do you actually think I would come in here if I didn't know?

Edited by ksoren
Link to comment
Share on other sites

Forget about AWE for a minute, okay? This has nothing to do with PAE. PAE just extends the PTE from 32 to 64 bits.
Correct, but the post was about moving a whole process VA into memory above 4GB on a non-Windows server OS, and to do so you *have* to have the PAE kernel loaded. AWE doesn't need PAE, it's just an API, but to use the RAM above the 4GB address limit you have to run the PAE kernel in Windows. That was my point. And technically, the overhead of PAE and doing the translation vs just using x64 just doesn't seem worth it unless you absolutely must use something that will not work on an x64 system.
Link to comment
Share on other sites

Yes, 32bit XP/Vista doesn't support >4G we can all agree on that. And yes, PAE is needed to address memory above 4G.

But your explanation of what PAE is and how it involves AWE is very flawed.

the overhead of PAE and doing the translation vs just using x64

This is why I mentioned the number of layers earlier. In both PAE and in 64bit mode the PTE is 64 bit wide. In PAE mode (which is default) the page table is 3 layers and in 64bit it is 4. So what has the most translation-overhead?

Edited by ksoren
Link to comment
Share on other sites

Well, that depends. I'm also using 32 more registers than I would on 32 bit, and my kernel pools are a lot less constrained (you'll use a greater percentage of your kernel pool on PTEs in x86 than you would in x64, even though the pool allocations are about the same), not to mention that if I have an app that can make use of the memory above 4GB natively (including the OS itself), it makes more sense to use the platform designed for it, not hacked for it. And although it's less of a problem nowadays, you still do have to worry about whether your x86 drivers will work properly on a PAE system vs using x64 drivers on an x64 system.

Obvious other benefits are LARGEADDRESSAWARE x86 apps getting up to 4GB VA on an x64 system, or being able to access 64 cores/processing units vs 32 with x86 (although client-side that's probably not an issue for years yet). Also, having greater than 8 GP registers is also a nicety for speed, plus allowing floating point operations to use SSE2 registers vs using the fp registers on x86, making use of fast calls and the ability to send 4 registers as an argument vs push'ing and pop'ing off the stack in x86 for fp, plus in x87 fp is stack-oriented, meaning more stack overhead and somewhat less efficient against the x64 design.

At least for Windows, unless you actually absolutely need x86, at this point, I recommend x64. Not just for large memory usage (although it is a plus considering buying 8GB of RAM is fairly cheap nowadays).

Link to comment
Share on other sites

Forget about AWE for a minute, okay? This has nothing to do with PAE. PAE just extends the PTE from 32 to 64 bits.

Um what? Excuse me. I think I choked when I read that.

First of all, Page Table Entries are mapped to Virtual Address spaces. So you're already wrong.

Now, Virtual Addresses are mapped to Physical Addresses and considering PAE stands for Physical Address Extension, you're wrong again

Let's add on the fact that PAE extends the physical address size from 32 bits to 36 bits and the Page Table Entry is extended to 64 bits to deal with PAE, you're wrong again. Like tic-tac-toe, three Xs, you're out.

Also only 51 bits are available for address bytes since 1 bit is used for the page size and 12 bits are used for operating system specific data or functions such as No eXecute.

Edited by jcarle
Link to comment
Share on other sites

Um what? Excuse. I think I choked when I read that.

Which part? AWE really has nothing to do with PAE. The PTE is 64 bit wide in PAE mode. Look in up. This doesn't mean all 64 bits are used for addressing memory (I didn't say that). Do you know what a PTE is?

First of all, Page Table Entries are mapped to Virtual Address spaces. So you're already wrong. Now, Virtual Addresses are mapped to Physical Addresses and considering PAE stands for Physical Address Extension, you're wrong again

I'm not sure what you mean. Yes, of course the page table is mapped into the VAS, otherwise the OS couldn't address it. I think you are confused about how paging works.

Also only 51 bits are available for address bytes

Last time I checked it was 52.

Edited by ksoren
Link to comment
Share on other sites

Anyways, intel.com and amd.com have some very good developer manuals on how it works inside the cpu.

Mark Russinovich is a popular guy. You could start with his Windows Internals book. He writes a bit about memory management in Windows. There are many other books on the subject.

Link to comment
Share on other sites

If you have a bicycle without handle bars, ...

Alright, I'm considering this hypothetical bicycle, go on ...

do the handle bars have nothing do with the bicycle and therefore the handle bars are just a way to drive the bicycle?

Uh ... what handle bars? :wacko:

Edited by seedofonan
Link to comment
Share on other sites

Uh ... what handle bars? :wacko:
I prefer to ride without handlebars and just go where the bike takes me. It's kind of like "I'm feeling lucky" in a Google search, but with a far bigger impact on the direction of my life. Yay chance!

/sarcasm ;)

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...