Corrected FDISK and FORMAT

[Guest wsxedcrfv]

Also BTW, if you give me MD5 hashes for those format programs, I may help you pinpoint where did they come from.

File Checksum Integrity Verifier (fciv.exe) version 2.05.

7f626f7a9eaa689e62faad9756d84695 format1.com 49,575 bytes 11-07-2006

75008d8a0e9e8b3004bba6a84640abd9 format2.com 49,415 bytes 06-07-2000

54c811ec0170a4c1743ebfb456748006 format3.com 49,655 bytes 05-05-1998

c95f2db4ad42536c0e1ebc4aeaab3ed7 format4.com 49,575 bytes 04-23-1999

945ac5c1fa9256c186eeebe48af16337 format5.com 49,543 bytes 05-01-1997

49a60d24e6dfd8f29798e152adbee6fd format6.com 22,670 bytes 09-16-1998

Edit: found 2 more (format5 and format6). Format6 comes from a rescue sub-directory contained inside Partition Magic v4.

Edited March 4, 2011 by wsxedcrfv

[Guest wsxedcrfv]

That squares with my own results: less than 450 GB are enough for them to give the "not enough memory" error. Since they also work OK up to 320 GB, there's just a 130 GB gray zone, between 320 and about 450 GB remaining to be explored for their true limit to be fully known. As I think I said elsewhere, the maximum cluster number accepted by scandisk and defrag (about 26,4 million clusters) would lead to a predicted maximum size (= 865 GB) much larger than that, so, while the maximum cluster number *is* a factor, it's not the only one that can prevent their correct working.

So given the following drives:

Drive C: 37,399 mb (37 gb), 4kb cluster size, 9.35 million clusters

Drive D: 1,430,447 mb (1.43 tb), 32kb cluster size, 45.8 million clusters

Drive E: 206,674 mb (206 gb), 32kb cluster size, 6.46 million clusters

Windows ME scandisk and defrag (scandskw.exe, diskmaint.dll, defrag.exe) function on Drive C and E, but give "not enough available memory" for Drive D.

Norton Disk Doctor (from Norton System Works 2002) gives a blue-screen fatal exception error halfway through analyzing drives C and D, but operates fine on Drive E. Even when the NOLBACHECK = 1 registry entry is used.

Windows 98se dos-mode scandisk (scandisk.exe) appears to operate just fine on Drive D.

Windows 98se unattended install works fine on Drive C.

Windows 98se DOS-mode format (format.com) correctly formatted Drive D as specified above after being partitioned with Free Fdisk.

I don't know if Windows 98se can correctly format Drive D (I might try that later). What I'll do now is use the DR Format and re-format Drive D using a smaller cluster size (maybe something crazy like 4kb) and then see if DOS-mode scandisk can take it.

Edit: ok, here's the deal with DR (Freedos) Format:

Using the command line "format d: /c:8 /u /q" should give me a 1.5 tb volume formatted with 4kb cluster size. DR format throws up all sorts of warnings saying the drive will not be compatible with windows 9x, yada yada yada. At the end of the operation, it reports that there are 366,144,578 allocation units, 1.43 tb total size, the volume serial number, and then follows that up with "Stack Overflow!". In fact, every variation I gave it resulted in the final output line saying "Stack Overflow!" after printing the volume serial number.

But what it seems to create is a volume that has a maximum of 45,779,966 clusters. It use the cluster-size that I tell it to use. So in this first case, when I say to use a cluster size of 4kb, the drive ends up having 45,779,966 clusters and an actual size of 183 gb.

If I tell it to use /c:16 (8kb cluster size), then it still ends up creating a volume using 45,779,966 clusters, 8kb cluster size, and the volume ends up being 366 gb in size. When I give it the command /c:64 (32 kb cluster size) then creates a volume with 45,774,322 clusters (which is what I expect).

So for some reason, 45,779,966 is a magic number of clusters for DR Format, and even when it thinks it's creating a volume with the correct number of clusters, it will ultimately not create a volume exceeding that number of clusters.

Edited March 4, 2011 by wsxedcrfv

[rloew]

FAT32 by design is limited to 28-Bitl Cluster Numbers. This will limit a FAT32 Partition to 1TiB with 4KB Clusters. At least 8KB Clusters are needed to properly Support 2TiB.

It may be possible to Format a larger Partition with 4KB Clusters, but it will get confused when more than ~1TiB of Data is written to it.

[Guest wsxedcrfv]

ok, here's the deal with DR (Freedos) Format:

Using the command line "format d: /c:8 /u /q" should give me a 1.5 tb volume formatted with 4kb cluster size. DR format throws up all sorts of warnings saying the drive will not be compatible with windows 9x, yada yada yada. At the end of the operation, it reports that there are 366,144,578 allocation units, 1.43 tb total size, the volume serial number, and then follows that up with "Stack Overflow!". In fact, every variation I gave it resulted in the final output line saying "Stack Overflow!" after printing the volume serial number.

But what it seems to create is a volume that has a maximum of 45,779,966 clusters. It use the cluster-size that I tell it to use. So in this first case, when I say to use a cluster size of 4kb, the drive ends up having 45,779,966 clusters and an actual size of 183 gb.

If I tell it to use /c:16 (8kb cluster size), then it still ends up creating a volume using 45,779,966 clusters, 8kb cluster size, and the volume ends up being 366 gb in size. When I give it the command /c:64 (32 kb cluster size) then creates a volume with 45,774,322 clusters (which is what I expect).

So for some reason, 45,779,966 is a magic number of clusters for DR Format, and even when it thinks it's creating a volume with the correct number of clusters, it will ultimately not create a volume exceeding that number of clusters.

Just to add a few more observations about DR Format: What I'm seeing in it's behavior is that when it's presented with, say, a 400 gb volume created with Free Fdisk, and instructed to format that volume using a 16kb cluster size, that what you end up with is a 200 gb volume with 16 kb clusters. In another example, when presented with a 600 gb volume created by fdisk, and instructed to format using 8kb cluster size, what you get is a 150 gb volume with 8kb clusters. After formatting, fdisk will identify these volumes as using 400 and 600 gb worth of hard-drive space, but DOS/Windows will only see these drives has having 200 and 150 gb of capacity. How this happens is unknown, but clearly DR Format has a problem correctly formatting large volumes with user-specified cluster sizes. It would seem that it first calculates the number of clusters a given volume should have based on a 32kb cluster size, then it formats that volume using that cluster-count but then sets the cluster size according to the user-specified command-line switch, resulting in a volume that's smaller in over-all size than intended.

The bottom line here is that we are still looking for a simple, working, DOS-mode Format program that allows the user to specify the cluster-size.

[Guest wsxedcrfv]

FAT32 by design is limited to 28-Bitl Cluster Numbers. This will limit a FAT32 Partition to 1TiB with 4KB Clusters. At least 8KB Clusters are needed to properly Support 2TiB.

It may be possible to Format a larger Partition with 4KB Clusters, but it will get confused when more than ~1TiB of Data is written to it.

Perhaps attempts to create a volume using more than 268 million clusters (2^28) will end up with a modulo number of clusters instead - regardless of cluster size?

[rloew]

FAT32 by design is limited to 28-Bitl Cluster Numbers. This will limit a FAT32 Partition to 1TiB with 4KB Clusters. At least 8KB Clusters are needed to properly Support 2TiB.

It may be possible to Format a larger Partition with 4KB Clusters, but it will get confused when more than ~1TiB of Data is written to it.

Perhaps attempts to create a volume using more than 268 million clusters (2^28) will end up with a modulo number of clusters instead - regardless of cluster size?

I don't remember seeing any mask applied to the number of clusters in VFAT.VXD so I think it will compute the actual number. The problem is that when you reach ~1TiB, the Cluster numbers will be confused with the Bad Cluster and End of Chain markers. I haven't analyzed to code to determine what happens after that.

My RFORMAT Program can create Partitions with any Cluster Size so I can run tests to find out more.

[Guest wsxedcrfv]

Well because I've been messing with this 1.5 TB drive along with trying different combinations of partitioning and formatting tools and posting those results here, and because I don't think we really know the upper limits of functionality when it comes to the Windows ME versions of scandisk and defrag (scandskw.exe, diskmaint.dll, defrag.exe) as transplanted into a win-98 system, I'll post what I've done so far.

I've downloaded an archived copy of Western Digital's Data Lifeguard for Dos software (dlgsetup11_dos.zip) - and for some reason the floppy image that it creates (using Caldera dos?) doesn't boot properly on my Asrock motherboard. So I then downloaded DR-DOS 7.01.08 (beta version - because the last stable version also wouldn't boot on my system) and I transplanted the DLG software onto the floppy with DR-DOS 7.01.08 and now I have a working bootable floppy of the WD tools.

I used that to create volumes of various sizes and cluster sizes on the WD 1.5TB drive, and then tested them in Win-98 running scandisk and defrag. I'm focusing first on using 4kb cluster size because I think that's the most interesting or useful case when running win-98 off large hard drives that have been partitioned into smaller drives.

I quickly determined that a volume with 32 "mega-clusters" (2^25) seems to be the functional cut-off for scandisk and defrag. (note - when I talk about scandisk here, I mean the GUI version of scandisk taken from Windows ME - which is scandskw.exe and diskmaint.dll).

When using 4kb cluster size, that works out to a volume size of 136 "billion" bytes, or close to 2^37 bytes.

I also determined that defrag has a slightly higher limit than scandisk, and that scandisk's limit is influenced by how many files (or how much capacity) is being used on the drive in question.

134,217,920 kb / 33,554,480 clusters / 137,439,150,080 bytes / no / no

133,408,220 kb / 33,352,055 clusters / 136,610,017,280 bytes / no / yes

132,606,536 kb / 33,151,634 clusters / 135,789,092,864 bytes / yes / yes

So what I have here is the total volume size (in kb) and # of clusters (as reported by chkdsk), total volume size (in bytes) as reported by windows properties, and the operational status of scandisk and defrag. In all cases, the drive in question was basically empty - other than the Recycled directory with is placed on it by Windows. While scandisk did run on the last line, it did not when I copied approx. 6 gb of files to the volume. Defrag did continue to be operational on that volume after the file-copy.

Previous reports have also indicated that win-ME versions of scandsk and defrag were functional at least up to 31.2 million clusters. I am also noticing that it takes DOS about 40 seconds to complete the first DIR command on the above volumes. I suspect that a volume formatted by Microsoft's format.com vs any other method will differ in whether or not there is a delay in handling the drive upon the first access. I believe this delay is also present when running windows 98 - in the form of a longer bootup time.

I may have to repeat some of these tests without having Norton SystemWorks installed, because the Norton Recycle Bin might be interfering with something here.

I'll do more testing later, but in the mean time please feel free to duplicate or replicate these results.

[rloew]

I am also noticing that it takes DOS about 40 seconds to complete the first DIR command on the above volumes. I suspect that a volume formatted by Microsoft's format.com vs any other method will differ in whether or not there is a delay in handling the drive upon the first access. I believe this delay is also present when running windows 98 - in the form of a longer bootup time.

The long delay running the DIR Command occurs when the Free Space entry in the Partition's Boot Record is marked Unknown. DOS or Windows has to read the entire FAT Table to compute, and set, this entry.

Not all Formatters set this entry to the actual Free Space.

[Guest wsxedcrfv]

I am also noticing that it takes DOS about 40 seconds to complete the first DIR command on the above volumes. I suspect that a volume formatted by Microsoft's format.com vs any other method will differ in whether or not there is a delay in handling the drive upon the first access. I believe this delay is also present when running windows 98 - in the form of a longer bootup time.

The long delay running the DIR Command occurs when the Free Space entry in the Partition's Boot Record is marked Unknown. DOS or Windows has to read the entire FAT Table to compute, and set, this entry. Not all Formatters set this entry to the actual Free Space.

That would explain a delay the very first time (and ONLY the very first time) you perform a DIR command after a volume has been formatted. It wouldn't explain a delay that happens the first time you perform a DIR command EVERY TIME the system is booted up.

And explain this:

The WD Data Lifeguard software is wizard-based. The drive formatting or preparation tool asks the following question before you can access the advanced formatting menu:

================

Select the Operating System below then select Next to continue:

Windows XP

Windows 2000

Windows ME

Windows 98se

===============

I found it strange that they would separate ME and 98se into separate options. If I recall correctly, when I did this a few years ago on some drives, the first few times I tried this I selected ME - but this gave some problem during installation of win-98 on the drive.

Are there differences in the FAT32 MBR or file structures or FAT tables between ME and 98se that would necessitate the formatting software know for which OS you intend to use on the drive?

Edit:

Ok, I've just made another observation here. Again using the 1.5 TB drive, I used WD-DLG to create a 500 gb volume using 32kb clusters. After booting back into DOS, the first command to access the drive (doesn't matter if it's DIR or CHKDSK) takes 20 seconds to complete. Until I re-boot again, every other access is done instantly. If I reboot, then again the first access takes 20 seconds.

Chkdsk says the new volume is:

531,308,448 kb total disk size

32 kb cluster size

16,603,389 clusters

When win-98 is started, scandisk and defrag (ME versions) have no problem with that volume. Norton Disk Doctor crashes the PC with blue-screen error.

Ok, so now what I do is boot into dos, and use win-98se dos-mode format.com on that volume. After it's done, I reboot - and a DIR or Chkdsk is performed instantly. No delays. I reboot several times and replicate this. Windows 98 seems to start faster too. Scandisk and defrag still work ok, and NDD still crashes.

So it seems that there are differences in the state of a volume caused by different formatting programs, and these states lead to at least one observable behavior - a delay (or absence thereof) in the first access to the volume during a usage or boot session.

The MS-DOS format program is obviously leaving the volume in a state that does not cause delays the first time the OS access the volume after booting. Even if the volume contains a rather large number of clusters (16 million, possibly more). This state persists long after the format procedure is performed. Clearly this behavior is desired - but how to achieve it with third-party drive preparation tools (or determine which tools also do this) would be useful, especially if a non-standard cluster size is desired (which format.com does not do).

Edited March 6, 2011 by wsxedcrfv

[rloew]

I am also noticing that it takes DOS about 40 seconds to complete the first DIR command on the above volumes. I suspect that a volume formatted by Microsoft's format.com vs any other method will differ in whether or not there is a delay in handling the drive upon the first access. I believe this delay is also present when running windows 98 - in the form of a longer bootup time.

The long delay running the DIR Command occurs when the Free Space entry in the Partition's Boot Record is marked Unknown. DOS or Windows has to read the entire FAT Table to compute, and set, this entry. Not all Formatters set this entry to the actual Free Space.

That would explain a delay the very first time (and ONLY the very first time) you perform a DIR command after a volume has been formatted. It wouldn't explain a delay that happens the first time you perform a DIR command EVERY TIME the system is booted up.

There may be a problem with the Information Sector (Second Boot Sector) that is preventing DOS or Windows from updating the Free Space Entry. You would have to send me the Second Boot Sector from both a normal Partition and from a Problem Partition to examine.

[Guest wsxedcrfv]

There may be a problem with the Information Sector (Second Boot Sector) that is preventing DOS or Windows from updating the Free Space Entry. You would have to send me the Second Boot Sector from both a normal Partition and from a Problem Partition to examine.

Ok. What I did was to capture the first 3 clusters of the volume, first with the drive formatted using format.com, and second with the drive formatted using the Western Digital Data Lifeguard program (WD-DLG). I captured the first 3 clusters because I wasn't sure if the 3 boot sectors are stored each in their own cluster. If they're not, then I suppose that the 3 boot sectors will all be found in the first cluster, and the other 2 clusters can be ignored.

I put all the files in mbr.zip, which can be downloaded here: <link removed>

But first a warning: Currently, fileden seems to have been hacked, and will try to serve you some malware (a browser add-on, followed by a pdf file). I don't think you'll get that if you try the direct link I gave, but you will get it if you just go to fileden.com.

I can tell you right now that there are some differences in the first sector between the two versions.

[dencorso]

I put all the files in mbr.zip, which can be downloaded here: <link removed>

But first a warning: Currently, fileden seems to have been hacked, and will try to serve you some malware (a browser add-on, followed by a pdf file). I don't think you'll get that if you try the direct link I gave, but you will get it if you just go to fileden.com.

Fileden is blacklisted by most antiviruses, at present, so that's a bad choice. You shouldn't have used it, in any case, after knowing there's malware around. So, please, do upload it again at a safer place. Try uploaded.to, if you want a suggestion.

[Guest wsxedcrfv]

Fileden is blacklisted by most antiviruses, at present, so that's a bad choice. You shouldn't have used it, in any case, after knowing there's malware around. So, please, do upload it again at a safer place. Try uploaded.to, if you want a suggestion.

Ok - I'll attach it here to this post.

MBR.zip

[dencorso]

Great. Thanks!

BTW, here's the identification I promised. I had posted it above, but moved it here, just to make sure you saw it.

File Checksum Integrity Verifier (fciv.exe) version 2.05.

7f626f7a9eaa689e62faad9756d84695 format1.com 49,575 bytes 11-07-2006 --> BHDD31E (Petr's)

75008d8a0e9e8b3004bba6a84640abd9 format2.com 49,415 bytes 06-07-2000 --> Win ME

54c811ec0170a4c1743ebfb456748006 format3.com 49,655 bytes 05-05-1998 --> chinese MS-DOS 7.1(?)c95f2db4ad42536c0e1ebc4aeaab3ed7 format4.com 49,575 bytes 04-23-1999 --> Win 98 SE 945ac5c1fa9256c186eeebe48af16337 format5.com 49,543 bytes 05-01-1997 --> Win 95 OSR2.5

49a60d24e6dfd8f29798e152adbee6fd format6.com 22,670 bytes 09-16-1998 --> Caldera OpenDOS 7.01

Format4 must have come from Win 98 SE (04-23-1999) and is binarily identical to the one in Win 98 FE (05-11-1998)

Format6 comes from a rescue sub-directory contained inside Partition Magic v4.

[Guest wsxedcrfv]

BTW, here's the identification I promised. I had posted it above, but moved it here, just to make sure you saw it.

Yea, so the format.com I've been using on the 1.5 tb drive comes from win-98se. Looks like it's good for the upper limit on a FAT32 volume - 2 TB yes?