Hard drives with 4kb sector-size are now available

[rloew]

I was able to decompress the Windows ME IO.SYS file and combine the data to produce an uncompressed IO.SYS file. It turned out that the entire file is decompressed in one operation. This made it a lot simpler.

Wow! That was fast! Way to go, RLoew! You do rock!

I presume you did it by hand... Please, do consider creating a decompressor tool to automate the process. Besides the default IO.SYS for hard disks and the one that comes in the EBD floppy, three other flavours of the Win ME IO.SYS are known: the one in the bootable floppy image contained in Win ME's CD, and those embedded in discopy.dll, retrievable with DiskExtract, thanks to Nuno Brito (there are two flavors of it, one in the discopy.dll from Win XP and another in the same dll from Vista or 2k3), not to mention those in KB311561... so that an automated solution would be a wonderful tool to have handy.

Done.

BTW, if I'm not mistaken, the compressed part starts with the identifier MSCM (which is reminescent of MSCF, the .cab file identifier), right?

Close, but no cigar. Even though next to each oter, the 'MS' and 'CM' as tested separately for different purposes. The 'MS' appears to be an integrity check. The 'CM' indicates the Compressed Data Header. It also appears at the start of the Decompression Code Header.

[dencorso]

Awesome!

[rloew]

Awesome!

I have published Version 1.0 of my Decompressor on my Website. It is in the Software Catalog under the Prerelease and Beta Section.

[Marius '95]

I have done some experimenting using a DDO that simulates a 4KB Logical Sector Hard Drive.

Sorry for the delayed response.

If you need to test different sector sizes for real (not using a DDO) I recommend using a SCSI HDD. SCSI HDDs support low level formatting with different sector sizes. You just need an old HDD (most 9 GB HDDs support this feature), an old SCSI adapter and Bart's SCSI Tool.

Run Bart's SCSI Tool and set a different sector size in drive geometry page, then low-level-format the drive. It takes 30min - 1h depending on drive speed and size. There is no progress indicator.

[rloew]

I have done some experimenting using a DDO that simulates a 4KB Logical Sector Hard Drive.

Sorry for the delayed response.

If you need to test different sector sizes for real (not using a DDO) I recommend using a SCSI HDD. SCSI HDDs support low level formatting with different sector sizes. You just need an old HDD (most 9 GB HDDs support this feature), an old SCSI adapter and Bart's SCSI Tool.

Run Bart's SCSI Tool and set a different sector size in drive geometry page, then low-level-format the drive. It takes 30min - 1h depending on drive speed and size. There is no progress indicator.

I am aware that SCSI Drives can be set for different Sector Sizes.

The DDO is more than adequate for testing DOS.

I had to add extra Patches to ESDI_506.PDR to simulate in 32-Bit mode but SCSI would not help since I could not use ESDI_506.PDR with it.

I was able to use a real 2KB Sector FileSystem by putting a FAT FileSystem on a CD.

[Ninho]

 There are no compatible biosses, so you won't be able to boot from a 4k sector disk.

This is plainly false. A standard AT BIOS will read one (1) sector from disk to a fixed place in memory, then jump to execute code from the start of that buffer provided the required signature, hex AA 55 is present at offset 510 (1fe hex). It doesn't matter if the sector length isn't 512, provided it is /larger/ (if it were smaller than 512, /then/ you'd have a problem because the boot sector signature wouldn't be found after reading the sector and so, BIOS would think there was a read error and the boot process wouldn't go on.

[rloew]

 There are no compatible biosses, so you won't be able to boot from a 4k sector disk.

This is plainly false. A standard AT BIOS will read one (1) sector from disk to a fixed place in memory, then jump to execute code from the start of that buffer provided the required signature, hex AA 55 is present at offset 510 (1fe hex). It doesn't matter if the sector length isn't 512, provided it is /larger/ (if it were smaller than 512, /then/ you'd have a problem because the boot sector signature wouldn't be found after reading the sector and so, BIOS would think there was a read error and the boot process wouldn't go on.

Reading the MBR would not be the issue. Unless the BIOS recognizes larger sector Hard Drives, it would only read 256 Words (512 Bytes) from the sector. This would leave the request incomplete with the drive waiting to return the rest of the sector. The drive would not be ready for the next request. Even if the BIOS managed to continue to boot by repeatedly resetting the drive between reads, it would fail as soon as the loader expected a larger sector.

[Ninho]

Reading the MBR would not be the issue. Unless the BIOS recognizes larger sector Hard Drives, it would only read 256 Words (512 Bytes) from the sector. This would leave the request incomplete with the drive waiting to return the rest of the sector.

Hmm, I suppose the low level BIOS routines would be accessing the HD controller in the old AT-compatible way, i.e. request a number of /sectors/ not /bytes/. But I haven't got 1024 byte per sector disks yet, so I could've held my mouth shut :~{ --- May I assume your reply was backed on actual trials you made ?

W/ best regards,

--

Ninho

Edited May 23, 2010 by Ninho

[rloew]

Reading the MBR would not be the issue. Unless the BIOS recognizes larger sector Hard Drives, it would only read 256 Words (512 Bytes) from the sector. This would leave the request incomplete with the drive waiting to return the rest of the sector.

Hmm, I suppose the low level BIOS routines would be accessing the HD controller in the old AT-compatible way, i.e. request a number of /sectors/ not /bytes/. But I haven't got 1024 byte per sector disks yet, so I could've held my mouth shut :~{ --- May I assume your reply was backed on actual trials you made ?

W/ best regards,

--

Ninho

Requests to Hard Drives are always based in Sectors. The Data transfer is in Bytes, Words or DWords. The BIOS would have to know the Sector Length to compute the number of transfer requests to perform or to tell the DMA controller to perform. The earliest versions of the protocol allowed specifying a Sector size, but I don't think it was ever used. The LBA protocol completely breaks this protocol. Sector size information was not added to the standard until very recently so only the newest BIOSes could possibly use it. Only by reading a Sector and monitoring the Data Ready Flag could an older BIOS determine the size of a Sector.

Since there are no ATA Hard Drives, that I know about, to test with, I could not run trials. As even the latest BIOSes I have seen, do not support the full 48-Bit LBA, but only 32 Bits, I doubt they would have Large Sector support.

Edited May 23, 2010 by rloew

[Ninho]

You're probably right - I was thinking of (very) old AT disks, chipsets and BIOSes - that did not use DMA for access to HD.

[rloew]

You're probably right - I was thinking of (very) old AT disks, chipsets and BIOSes - that did not use DMA for access to HD.

Even PIO transfers will not work properly unless you specifically check for Sector size. In PIO mode this is not hard, but the BIOS must be coded for it.

[billtodd]

Is it possible just to choose appropriate partition starting locations to get pre-Vista systems to work well with 4 KB sector drives, as long as they emulate 512-byte sectors (as I'd kind of expect them to do for a long time, at least as an option, if disk manufacturers don't wish to kiss the legacy aftermarket good-bye)?

E.g., for the normal case where traditional partitions are based on fictitious 'cylinder' boundaries (512-byte sectors plus the fictitious 255 heads and 63-sector tracks - i.e., 16065-sector 'cylinders' each a bit under 8 MiB; old Thinkpads and perhaps other systems used 240 fictitious heads, so would need to be handled differently), you'd just define a 62+ MB initial partition to create a suitable end-point, then create the actual desired partition after it with its start point a multiple of 4 KB from the start of the disk (then delete the preceding small partition if if you needed 3 more primary partitions).

At least if you were creating a partition for a Win2K or later NTFS system, where (IIRC) the file clusters (if a multiple of 4 KB themselves) begin at 4 KB multiples from the partition's start (yes, that offset may be configurable in the Volume Boot Record, but just in case some brain-damaged software is ignoring that it seems best to leave it 'standard').

But in FAT partitions (in all Windows environments, or just in Win9x?) my vague recollection is that clusters (even if a multiple of 4 KB themselves) are not so nicely aligned with respect to the partition's start. Perhaps RLoew's format utility changes the Volume Boot Record and first-cluster offset to fix this - but if one knew the 'standard' offset (if there is one, you could simply create a FAT partition with 4 KB clusters and see what the alignment was of some file in it) you could, again, just adjust the partition's start location to make the file clusters fall on disk 4 KB boundaries.

Logical partition placement is complicated by the existence of an Extended Partition Boot Record (one fictitious track, like the MBR 'track') embedded in but not actually part of the associated logical partition (so you'd need to have the preceding partition end on a 'cylinder' boundary that's 63*512 bytes before a 4 KB boundary to leave room for it - again, with a partition file system that maintained 4 KB cluster alignment internally, so adjust differently for FAT). Still, eminently feasible.

As I noted above, one of the reasons I prefer this approach is because it does nothing unusual that might cause traditional disk utilities to screw things up. For example, using pre-Vista utilities (including Windows Disk Management itself) on new-style Vista/Win7-partitioned disks can cause data loss (documented by Microsoft, though I don't have the link at my fingertips), as has use of Vista/Win7 Disk Management to alter the layouts of traditionally-partitioned disks (this last can be fixed by modifying Vista/Win7 Registry values to make them use the traditional partitioning layouts, but the former cannot).

So if offset-to-first-cluster is a standard value I'd prefer this approach on any disk where I thought I might wish to operate on it with pre-Vista partitioning software (which is still true for most disks I use, given my fondness for multi-boot environments). Any insights into whether I'm overlooking something significant here?

Edited June 1, 2010 by billtodd

[jaclaz]

As I noted above, one of the reasons I prefer this approach is because it does nothing unusual that might cause traditional disk utilities to screw things up. For example, using pre-Vista utilities (including Windows Disk Management itself) on new-style Vista/Win7-partitioned disks can cause data loss (documented by Microsoft, though I don't have the link at my fingertips), as has use of Vista/Win7 Disk Management to alter the layouts of traditionally-partitioned disks (this last can be fixed by modifying Vista/Win7 Registry values to make them use the traditional partitioning layouts, but the former cannot).

Yep, just for the record:

http://www.boot-land.net/forums/index.php?showtopic=9897&hl=

if you happen to find the actual link on MS, I would appreciate it.

jaclaz

[billtodd]

if you happen to find the actual link on MS, I would appreciate it.

jaclaz

I read only the first page of your link above, but it did contain a link to multibooters.co.uk which contains a link to http://support.microsoft.com/kb/931854 (describes only losing a Vista partition after using XP to add a new one, but I'm pretty sure that there's another somewhere that describes - IIRC - losing all old-style logical partitions after a new one created with Vista/Win7).

[jaclaz]

I read only the first page of your link above, but it did contain a link to multibooters.co.uk which contains a link to http://support.microsoft.com/kb/931854 (describes only losing a Vista partition after using XP to add a new one, but I'm pretty sure that there's another somewhere that describes - IIRC - losing all old-style logical partitions after a new one created with Vista/Win7).

Yep , just asking if there is anything else.

You might want to notice how the KB931854, besides the utter unusefulness of the proposed workaround:

To work around this problem, do not use Windows XP to create a partition on a computer that also has Windows Vista installed.

as well as the incipit:

If you use Microsoft Windows XP to create a new partition...

are WRONG!

What edborg reported in the linked to thread is that the same happened by ONLY changing the Active status of an existing partition , in other words in order to change a single byte from 00 to 80 or viceversa the stooopid XP disk management created havoc!

jaclaz