System Files UPX-izer.

[Francesco]

I was thinking about creating an utility that compresses with UPX all windows system files (all the EXEs and DLLs present in the XP cd). Usually UPX create smaller executable/dll files that execute/load faster.

Some of those EXE/DLLs give problems when UPX-ized but if somebody is interested in this project we can try finding out the files that can be UPX-ized properly.

This not only would make windows installs smaller but it should also make them faster (obviously after we can get windows working with as much possible compressed files we can do some benchmarks). This will affect only applications loading time, not applications speed.

Is anybody here interested?

Edited October 13, 2006 by Francesco

[jdoe]

@Francesco,

It would make the Windows install smaller but surely not faster. Compression means decompression and it take extra time for PE to loads so the OS will be SLOWER.

[jaclaz]

@jdoe

This is not necessarily true, given the relatively slow transfer speed of disks when compared to the relatively high speed of today's computer processors, the "break even point" must be calculated.

The real problem with UPXed files is the amount of RAM that is needed to run them, and a general form of "dirtying" memory pages, that could lead to instability problems, you will find this small article interesting:

http://f0dder.reteam.org/packandstuff.htm

@Francesco

Sorry to say so, but your idea is not really new:

http://www.msfn.org/board/index.php?showtopic=4120

the problem, you see, is that not all system files CAN be compressed, as some, apart from the considerations above exposed, won't simply work.

Some work, with regards to a limited sub-set of XP files has been done in the Winbuilder (was Openbuilder) project:

http://boot-land.net/forums/index.php

http://www.boot-land.net/forums/index.php?&showforum=22

but you will understand that extending that to the hundreds of files that are part of a full-blown XP is quite a HUGE amount of work, even because some files are not normally used and an eventual bug (compressed file that should have not) would take months to be discovered.

Moreover, during development of UPX scripts in the said prohects, it has become eveident that some Windows 2003 files behave differently form XP SP2 one, it is possible that newish releases (Windows Update) of a .dll that was in the earlier version "UPXable" are not anymore.

jaclaz

[jdoe]

@jdoe

This is not necessarily true, given the relatively slow transfer speed of disks when compared to the relatively high speed of today's computer processors, the "break even point" must be calculated.

The real problem with UPXed files is the amount of RAM that is needed to run them, and a general form of "dirtying" memory pages, that could lead to instability problems, you will find this small article interesting:

@jaclaz,

My post was not technically strong as yours. You did a good search.

My post only refer to a test I did long time ago about the idea of using UPX to compress Windows Portable Executables and it was leading to a considerably slow OS. I don't have any technical information to prove my point but I know that in a real test, I got that result.

[jaclaz]

I don't have any technical information to prove my point but I know that in a real test, I got that result.

Yep, it is also possible that on a system with a relatively small amount of RAM, disk swapping increased, and that would reduce system "responsiveness" to a crawl.

Just to give you an example I tried once the same "reduced", normal and UPXed windows 2K setup on three machines:

1) VIA EPIA with 533Mhz Processor, 256 Mb RAM, ATA 66 drive

2) VIA EPIA with 1000Mhz Processor, 512 Mb RAM newish (at the time) ATA100 drive

3) "standard" ATX with 2,8 Ghz AMD Processor, 512 MB RAM, ATA 133 Drive

on 1) the UPXed version was slowish once a few apps were loaded, most probably for the combined effect of CPU overhead and to the HD swapping (not enough RAM)

on 2) the speed difference was negligible, could not distinguish between the two

on 3) the UPXed version was apparently a bit faster

I did not actually made any measurements , the above were just my "feelings", at the time what I was interested into was to build a minimal Win2K that could fit in a ZIP disk, the "final" version running from ZIP disk, proverbially slow, was definitely faster in the UPXed version.

I wonder how this would play with XP on newish fast processors and SATA drives....

jaclaz

[Francesco]

@jdoe

This is not necessarily true, given the relatively slow transfer speed of disks when compared to the relatively high speed of today's computer processors, the "break even point" must be calculated.

The real problem with UPXed files is the amount of RAM that is needed to run them, and a general form of "dirtying" memory pages, that could lead to instability problems, you will find this small article interesting:

http://f0dder.reteam.org/packandstuff.htm

@Francesco

Sorry to say so, but your idea is not really new:

http://www.msfn.org/board/index.php?showtopic=4120

the problem, you see, is that not all system files CAN be compressed, as some, apart from the considerations above exposed, won't simply work.

This is why I said that I would have needed help of other people to identify the files that can't ee compressed.

BTW that url is about mad cows.

Some work, with regards to a limited sub-set of XP files has been done in the Winbuilder (was Openbuilder) project:

http://boot-land.net/forums/index.php

http://www.boot-land.net/forums/index.php?&showforum=22

but you will understand that extending that to the hundreds of files that are part of a full-blown XP is quite a HUGE amount of work, even because some files are not normally used and an eventual bug (compressed file that should have not) would take months to be discovered.

Usually DLLs and Executable that shouldn't have been compressed don't even load so it shouldn't have took a long time to identify them.

Moreover, during development of UPX scripts in the said prohects, it has become eveident that some Windows 2003 files behave differently form XP SP2 one, it is possible that newish releases (Windows Update) of a .dll that was in the earlier version "UPXable" are not anymore.

Obviously the system files compressor would have checked the files MD5 before compressing to identify if the file is the right file to patch.

However if you say that it's not worth it because it needs more RAM and applications may run slower I will trust you. I've only seen applications launching faster but I never checked out the ram usage and I've never benchmarked them.

Edited October 13, 2006 by Francesco

[LLXX]

The 'proper' (or, according to English syntax) noun should be "UPX'er". Files packed with UPX should be referred to as "UPX'd". Here:

for %a in (*.dll *.exe) do upx %a

That'll UPX all the files in the directory.

Of course, you should do this from a DOS boot or alternate Windows install, as it can't compress live files like the kernel and shell.

Basically, whether or not you see a speed improvement (or degradation) depends on the ratio of CPU speed to I/O speed. Compressed files will load faster if the I/O from the hard drive is taking the bulk of the time, whereas they'll load slower if the HDD is fast and the only overhead is the decompression.

Compressed DLLs cannot be shared between processes, so memory usage increases linearly with the number of running processes using that DLL.

Edited October 13, 2006 by LLXX

[jaclaz]

BTW that url is about mad cows.

I beg your pardon?

jaclaz

[Francesco]

BTW that url is about mad cows.

I beg your pardon?

jaclaz

http://www.msfn.org/board/index.php?showtopic=4120 (the url you posted) is really about mad cows. I'm not joking.

[bledd]

lol, i was thinking that too

[LLXX]

Either they reorganised the topic #s or a digit was deleted during posting.

[Francesco]

Either they reorganised the topic #s or a digit was deleted during posting.

Let's hope that no mad cows were harmed during the process!

Edited October 15, 2006 by Francesco

[jaclaz]

Yep, the intended link was this one :

http://www.msfn.org/board/index.php?showtopic=41208

Sorry for the mis-copying.

However, I quicly contacted the International Society for Cow Protection:

http://www.iscowp.org/

and they assured me that NO cow, mad or sane, was actually harmed in the process, so everything should be well now.

jaclaz