Vista X64 SP1 RC .... weird memory footprint "drop" ?

[benifin]

Hi all

As slipstreaming is too much of a pain until Vista SP1 goes RTM - by which stage no doubt Vlite will allow for integration - I have done as follows:-

- full clean Vista X64 install - no removals at all / no vliting at all - runs perfect

- full clean SP1 RC update - runs perfect

- set PageFile set to 3gig - runs perfect

- I then disable ALL Scheduled Tasks and disable all but the critical 10 Services - runs perfectly

Anyway, as I said, I do the above and all works perfectly .............. except for one bizzare thing ......

=> I boot up

=> system goes into Vista SP1 RC GUI no problems

=> Ram foot-print / useage is ~850 meg

=> I let the system sit there and idle

=> after 2 min 30 sec the Ram foot-print DROPS in one hit and in an instant from ~850 meg to ~350 meg

This is fully repeatable and happens every single time.

So, my question is:-

- does anyone have any idea what component is most likely causing this Ram behaviour ?

Thanks,

Ben

[aviv00]

does superfatch is runing ? or ReadyBoost

[benifin]

both off - am only running the basic / core system services needed to boot and run Vista - everything else is disabled ...... thats why I figure it must be a specific and large component ....

Ben

[aviv00]

maybe u could use process explorer to figure it out

save the list of process after and before and maybe u will see the diff [hope so]

[nuhi]

Try to disable Windows Search service if you haven't. But since you say all are disabled than that proves to you that it's not the same as removals.

[benifin]

Try to disable Windows Search service if you haven't. But since you say all are disabled than that proves to you that it's not the same as removals.

Hi nuhi

Windows Search is disabled - as are all other non-core-system services - as you say, disabling system services and disabling scehduled tasks definetly does NOT equal removal - thats the single best thing about Vlite. You can tweak Vista all you want but nothing you do will ever be as efficient as removing the actual components.

nuhi .... beacuse its ......a single massive ram footprint drop of ~500meg ..... and becauses its fully repeatable ........ and beacasue it happens at the same time after boot up every time ....... and because it happens instantaneously and ......... the system is fuly idle ................... is there any specific component [or group of components] that spring to mind that may be causing this ?

Reason I ask is that I'm thinking if there is a likely culprit, I might try Vlite'ing the X64 install before installing SP1 RC to see (i) if it still installs SP1 RC with this component removed and (ii) if it does, does it "overcome" this ram footprint dropping issue.

Thanks

Ben

Edited December 17, 2007 by benifin

[nuhi]

Could be simply cache (from basic prefetch or simply windows starting?) being released if not used, wouldn't bother with it. If you ask me fill the cache to the top (as Vista does while used) and drain it only when necessary, why use hdd more than needed.

[dexter.inside]

=> after 2 min 30 sec the Ram foot-print DROPS in one hit and in an instant from ~850 meg to ~350 meg

This is fully repeatable and happens every single time.

So, my question is:-

- does anyone have any idea what component is most likely causing this Ram behaviour ?

Thanks,

Ben

Most people are not yet aware of the new behavior the NT 6.0 Paging Executive operates - I'd recommend the new reviews by Winternals on Technet covering the 6.0.5384+ kernel behavior.

NT 6.0+ kernels operate like this with physical memory:

- windows startup, load prefetch boot apps in physical memory

- at least 10% free physical memory? if yes, load rest of prefetch app list in physical memory

- user logon

- Is SuperFetch enabled? if yes, load superfetched list of apps in physical memory

- Is ReadyBoost enabled? if yes, use ReadyBoost allocation on usb disk to stripe with pagefile (like RAID-0 with stripe)

- at least 10% free physical memory? if yes, use it for other system cache purposes

- a running process ends, keep it loaded in memory for quick access

- a new process starts:

- is it superfetched? pass execution to superfetched code in physical memory buffer

- is it prefetched? if yes, is it in the pagefile? if yes, page it in. pass execution to it.

- if not prefetched or ReadyBoosted, load from disk or from disk cache

- at least 10% free physical memory? if not, page out allocated memory to swap, until 50% free.

- allocate memory for app

Note: superfetched code is executable (but Data Execution Prevention on /optout will disable it) and readyboosted code is not executable (NX/XD bit prevents its execution) and must be copied / paged in physical memory.

What you've noticed happens in these cases:

- app is started, is not cached, and less than 10% and/or less than app requirement for free physical memory is available

- user account is locked (triggers ~50% physical memory to be paged)

- memory recycler (system cache flush, or on demand, like from TrustedInstaller)

- switching to secure desktop when less than 10% available (like starting UAC or CTRL+ALT+Del)

Edited December 20, 2007 by dexter.inside

[nuhi]

dexter.inside, yeah exactly. What do you think that breaks when turning off UAC from those on the list, maybe Prefetch?

benifin, you could do a test by leaving UAC enabled and empty the C:\Windows\Prefetch or search for a tweak how to disable it, then compare.

[dexter.inside]

dexter.inside, yeah exactly. What do you think that breaks when turning off UAC from those on the list, maybe Prefetch?

benifin, you could do a test by leaving UAC enabled and empty the C:\Windows\Prefetch or search for a tweak how to disable it, then compare.

Turning off UAC just wastes some memory with prefetching... Paging Executive will keep probably ~4 MB of code (flagged as non-pageable) all the time in memory for some of the DLLs that UAC requires. The tradeoff is that you won't get the Secure Desktop (Session ID 0) prompts anymore while you work and your memory won't get paged out in the middle of your activities.

Which by the way is exactly the reason for UAC not being enabled by default on Administrator. As for accounts without administrative privileges, it seems Microsoft has a different opinion than most of us of what users usually are allowed to do.

Why do you think disabling UAC breaks something?

[nuhi]

What?

The point is even if you disable all services and turn off UAC (not disable the driver) the memory will be filled and released after some time. But if UAC is disabled in the driver the memory won't be filled in the first place. Which brings me to the conclusion that it may be that prefetch fails and doesn't load cache or how do you explain 450MB in RAM not being loaded?

Remember in XP if you remove Task Manager prefetch fails.

Maybe you missed the other topic:

http://www.msfn.org/board/index.php?showtopic=109402

Disclaimer:

I haven't confirmed this memory difference nor tried.

[dexter.inside]

I think this is related to the luafv (Microsoft Load Utility Agent - File Virtualization Driver). I have no idea how files are virtualized with it, but the driver development kit points out that it works by keeping a process in memory and generating new instances that point to that buffer for each session id that requests the file.

As expected, its public interface is undocumented. Ryan Price and Chris123NT have already beaten me to it and pointed out the bug in MS Connect for SP1 RC1:

When UAC is disabled, this process can not function properly and when booting up directly affects the boot process. Whenever UAC was enabled, boot times seemed to improve; understandable because the driver is able to load without any problem. Disabling UAC suddenly caused the boot up time to pick up 20-30 seconds on average. After going into the registry Chris and I found a key which corrisponded to the driver loading at start up. Located at: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LUAFV

REG_DWORD "Startup" - The default is 2 but setting it to 4 disables it completely.

This has cut down boot times significantly in some cases. The suprising thing is this is across Vista RTM and the current beta build of Vista SP1 (6001.17042). Microsoft should patch this so that as someone disables UAC it also disables this driver from loading so that the user does not take a performance hit for disabling UAC.

Nuhi, 2 questions:

1) is there anything in vLite that removes luafv and/or dependencies from WinSxS or registry?

2) is it a good idea when you choose to disable UAC from vLite to add this registry tweak?

From what I gather, I thing luafv works like this:

- Boot process reads layout.ini in \WINDOWS\Prefetch and allocates a luafv buffer in memory that contains all the .pf files (note that ntoskrnl.exe is also prefetched in NTOSBOOT-XXXXXXXX.PF)

- Is luafv buffer valid? generate luafv instances for session id 0, else skip prefetch

- NT 6.0 starts loading (either from luafv instances to prefetch items, or from disk reads)

- At user logon, luafv is invoked to generate instances to pretetch items for session id 1 to n (in theory this should save memory)

The problem outlined by this topic could be the luafv buffer wasting memory. Prefetch could be creating the buffer, but if there's no luafv, it will remain stuck and eventually paged out (as there won't be any code to create instances for session ids and hook them to that memory area)

Note that luafv buffers are pageable, even if they contain code marked as nonpaged, which seems quite strange, as you can create a nonpaged luafv buffer instance to a piece of pageable memory buffer. This may be useful for a Terminal Server on Server 2008, but looks like crap to me on Vista. Has anybody seen less nonpaged kernel memory when prefetch works? This may help point out if prefetch works or not. Hope to see the specs before 2009 on MSDN.

Happy testing!

Edited December 20, 2007 by dexter.inside

[nuhi]

Interesting, so the conclusion is that prefetch is disabled if UAC is turned off but the driver is enabled? Is it reenabled when the driver is disabled? That kinda doesn't go in favor of his 450MB difference because prefetch when enabled should use more memory and boot SLOWER? (because it preloads more)

1. only in newer 1.1 RC when it comes out

2. Turn off UAC tweak has the same effect as the cpanel - users setting does, meaning it doesn't touch the luafv start setting

My guess, as flawed as it may be I'll just say it again, is that luafv disabling kills prefetch and that makes Vista spend less memory and boot faster, no preloading.

Aand how to test for this? What about Superfetch, does it rely on Prefetch...wiiiii so many questions. Lets remove it all and not worry

[dexter.inside]

Superfetch is probably using task scheduler to handle buffers either through luafv or directly with the filesystem.

Yes, it's a good idea to remove them all and try to quantify speed differences on the same machine.