The smallest possible size of boot.sdi

[joakim]

I've been annoyed by the +3Mb size of the supplied boot.sdi that is shipped with nt6.x. It is really a size overkill! My tweaked boot.sdi is now at 960 kb, or 983 040 bytes to be exact. The boot.sdi is basically an sdi with just a tiny ntfs partition image injected and a marker for the "mysterious" wim blob at the very end. Download at; http://www.mediafire.com/file/t0zzwmwwylb/boot_sdi.zip

If anyone can reduce the size further, I would be happy to inspect the file.

Why on earth did MS make it so big? Only answer can be that they are lazy and don't care about a few MB of wasted space

Joakim

Updated version 300 kB; http://mft2csv.googlecode.com/files/boot_sdi_300.zip

Details of sectors inside the latest partition image;

$Boot 0 
$MFT 1-65 
Root directory 66-73 
$LogFile 74-586 
$MFTMirr 136-143 
$UpCase 136 
$Secure:$SDS 137-392 
$Bitmap 138 
$AttrDef 139-143 

The rest of the systemfiles are fully contained within $MFT

The reorganization of the ntfs metafiles was done to make it easier for those that want to decode and understand it.

Edited August 27, 2012 by joakim

[jaclaz]

NTFS is a rather "bulky" filesystem for very little sized volumes.

Strip from the file first 0x2000 or 8192 bytes and save as bottsdi.raw.

Open this file with 7-zip and you will see the NTFS filesystem structures that occupy a lot of space.

The problem was talked about many years ago when Mark Russinovich made possible to have NTFS formatted floppies:

http://freewareapp.com/ntfsflp_download/

http://web.archive.org/web/20000511191617/www.sysinternals.com/ntfsflp.htm

http://web.archive.org/web/20000511191617/http://www.sysinternals.com/ntfsflp.zip

It seems like the Log file can be reduced from the current 524,288 bytes to 262,144, that at the time was found to be the bare minimum.

Compare the NTFSIMG in the download with the bootsdi.raw file in 7-zip .

jaclaz

[joakim]

Cool, I did not know. Will investigate it soon and see what we get.

[joakim]

Here is a new one at 696 kB; http://www.mediafire.com/file/5p9nsoep8o7f5ib/boot_sdi_696.zip

It is a modified version of Mark's ntfs partition. The partition image is only 708 608 bytes.

I wonder why Microsoft made their boot.sdi almost 4,5 times the size of what is necessary? It is really just a waste of space..

Joakim

Edited July 18, 2010 by joakim

[joakim]

Last shot at 692 kB; http://www.mediafire.com/file/xq03xipghtjns31/boot_sdi_692.zip

By shrinking the metafile $Boot from 8 192 bytes to 512 bytes, the total partition size is now 684 kB or 700 928 bytes. The partition has no free space and only contains ntfs systemfiles. It will only work for booting nt6.x based winpe (does not write anything to the partition, just mounting the wim). I doubt it is possible to reduce the size of it any further. Partition image is also included in download.

Joakim

[joakim]

Size of boot.sdi is now 319 488 bytes: http://www.mediafire.com/file/cvabeux4rj6xri6/boot_sdi_312.zip

The interesting stuff is that some metafiles only need to have some of the first bytes present. And some of them are not read at all. But since their reference can't be removed from $MFT, their occupied sectors can be filled with 0's. Namely $MFTMirr and $Bitmap are not read at all. $MFT and $AttrDef are not modded at all. For $UpCase only the first sector needs to be present. As mentioned before, the $Boot can be reduced to 1 sector (IBL not needed). The $LogFile only requires the first 8 sectors to be present (rest can be 0's). However the size of the LogFile can't be reduced from its current size of 262 656 bytes. Now the most interesting bit is that some metafiles can be cross referenced, meaning they can occupy the same sectors. This will be true when certain sectors are not read, or their content are the same (like with the 0's). In this boot.sdi, as much as 3 metafiles are located on some common sectors. The $Secure metafile effectively only takes 1 sector (rest can be 0's). It is entirely located inside $LogFile, and because $LogFile can't be shrinked, I did not bother much about its size.

If the $LogFile can be reduced in size, it is likely that the total partition image size can be reduced to the crazy size of 73 216 bytes!!

This must only be considered as research and a PoC. It works, but has not been extensively tested.

Joakim

[jaclaz]

VERY GOOD WORK!

About this:

I wonder why Microsoft made their boot.sdi almost 4,5 times the size of what is necessary?

There is an easy answer:

This is by design.

http://www.boot-land.net/forums/index.php?showtopic=3541

A seemingly OT (but not much) idea:

Is there any real *need* for the filesystem inside the boot.sdi to be NTFS?

Or FAT 12 would do?

jaclaz

Edited July 27, 2010 by jaclaz

[joakim]

Is there any real *need* for the filesystem inside the boot.sdi to be NTFS?

Or FAT 12 would do?

According to their own doc, it must be ntfs for rw's; http://technet.microsoft.com/en-us/library/cc722145(WS.10).aspx

You can mount a .wim file with read/write permissions only on an NTFS file system.

Joakim

[jaclaz]

Yep, I completely missed that it was used as a mounting point.

But in the original use the .wim is not Written, is it?

Still I don't think you can mount anyhting on FAT at all.

I guess that this is - as often happens - a bit misleading:

http://technet.microsoft.com/en-us/library/cc722145(WS.10).aspx

You can mount a .wim file with read/write permissions only on an NTFS file system. This avoids the 2 gigabyte (GB) barrier that is imposed by FAT file systems and prevents data loss that is possible with FAT or other non-NTFS file systems.

jaclaz

Edited January 18, 2011 by Tripredacus
fixed link

[joakim]

But in the original use the .wim is not Written, is it?

Still I don't think you can mount anyhting on FAT at all.

That's right. It will bsod 0x..ED with a fat partition.

Joakim

[joakim]

Size now at 307 200 bytes; http://www.mediafire.com/file/k3qdkrucb6ce8ej/boot_sdi_300.zip

The partition image is now basically constructed like this;

Sector 0:         $Boot
Sector 1 - 64:    $MFT
Sector 65:        $MFT:$Bitmap
Sector 66 - 73:   Root directory
Sector 74 - 586:  $LogFile

The rest of the systemfiles are found "inside" the $LogFile starting at sector 136.

It is now much easier to read it too.

Also tried putting the $LogFile in sector 1 and placing all other systemfiles inside it, but that did not work.

Btw, it is a really good way to learn about ntfs when working with such a small partition image.

Joakim

Edited July 28, 2010 by joakim

[jaclaz]

Good.

Now it comes to mind a question:

if a single guy in a few days of his spare time can take a "random" item and reduce it's size tenfold what could do the full-time MS guys if they wanted to?

As a dinosaur, in my simplicity I continue thinking that smaller things are faster, no matter how faster is your hardware, managing less bytes it will make it faster!

jaclaz

Edited July 28, 2010 by jaclaz

[Kelsenellenelvian]

As a dinosaur, in my simplicity I continue thinking that smaller things are faster, no matter how faster is your hardware, managing less bytes it will make it faster!

AMEN!

[wimb]

Now it comes to mind a question:

if a single guy in a few days of his spare time can take a "random" item and reduce it's size tenfold what could do the full-time MS guys if they wanted to?

Reduction of Windows 7 size by factor 40 would be possible and interesting .....

Functional Size can be around 200 MB .....

[Tripredacus]

As a dinosaur, in my simplicity I continue thinking that smaller things are faster, no matter how faster is your hardware, managing less bytes it will make it faster!

AMEN!

This exact idea is also present in the web design community. Us "old schoolers" were taught that you needed to make your code as small as possible so that the page would load fast because most people had dial-up. Now that most people have broadband, no one cares that their code is all bloated because no one is going to notice.