Windows 98 and the 4gb file-size limitation

[Guest wsxedcrfv]

Some of the blue-ray movie downloads I'm doing are resulting in files larger than 4gb, so it would be somewhat useful to break the 4 gb barrier. The volume that windows 98 itself is installed on would not need this capability - only a second slave drive would need this (likely SATA).

It's my impression that the 4 gb file-size limitation is rooted in the FAT32 specification and not necessarily with win-98 itself.

Is it possible to use an alternate file system (some modification or enhanced version of FAT32) that exceeds the max-file-size limitation of 4gb of FAT32 under Windows 98se - or perhaps even Windows ME?

Edit:

Alright - never mind. I was just reading this:

I wonder if some modification of FAT32 is possible to break the 4 gb file-size limitation. Like maybe doubling the cluster-size to 64kb. Basically, create some new version of FAT32? For example, I see that there are 4 bytes allocated for the file-size in the directory table. Who says it can't be 5 bytes? That would code for an impossibly large file size.

Edited March 25, 2011 by wsxedcrfv

[dencorso]

Yes, the 4 GiB (-1 byte) limit is intrinsic to the design of the FAT filesystem (no matter whether 12, 16 or 32, but only FAT-32 has volumes big enough to hit it), since it's due to representing the file size, in the directory structure, as an unsigned double-word (= 32 bits), resulting in a [(2^32)-1] max value, since the counting is from zero.

No, even if you use UDF or NTFS, Win 9x/ME internal structures also use 32 bits numbers internally, so the OS itself is limited to a 4 GiB (-1 byte) max file size, and only a huge amount of careful patching might solve it. So, for all practical purposes, no, it cannot be done. RLoew and Xeno86 confirmed this in this thread of not very long ago. Nothing changed, nor is likely to. Of course, an ExFat driver may be written, but it'll also be limited to 4 GiB (-1 byte). Sorry.

[Guest wsxedcrfv]

I suppose the solution to the 4 gb problem is this:

Build a new PC with win-2k or XP and use it only as a file-server and attach it to your home or soho network and from your win-98 machines map a network drive to the NT machine and keep your large files on the networked drive. Winrar running on win-98 ought to be able to save it's decompression output to the networked drive, and VLC running on win-98 ought to be able to play large movies also from the networked drive.

[rloew]

I suppose the solution to the 4 gb problem is this:

Build a new PC with win-2k or XP and use it only as a file-server and attach it to your home or soho network and from your win-98 machines map a network drive to the NT machine and keep your large files on the networked drive. Winrar running on win-98 ought to be able to save it's decompression output to the networked drive, and VLC running on win-98 ought to be able to play large movies also from the networked drive.

Won't work. I tried that a long time ago.

Windows 9x IOS uses 32-Bit File Pointers, so the Pointers are truncated before they even reach the FileSystem Drivers. This includes the Network Redirectors.

[Guest wsxedcrfv]

Winrar running on win-98 ought to be able to save it's decompression output to the (NTFS) networked drive, and VLC running on win-98 ought to be able to play large movies also from the networked drive.

Won't work. I tried that a long time ago. Windows 9x IOS uses 32-Bit File Pointers, so the Pointers are truncated before they even reach the FileSystem Drivers. This includes the Network Redirectors.

Well, I can confirm that winrar running on win-98, decompressing a multi-part RAR file that is larger than 4 gb to a network drive on a machine running NT4-server, will NOT display any error message during the decompression, and the decompression will appear to run fine all the way to completion, but the resulting file sitting on the NT4 server will be exactly 4.00 gb (4.2-something billion bytes). In this case, an mkv file that I could not get VLC to play. (I assume that NT4 does not have a 4 gb file-size limitation... ?)

What I don't understand - do you really need a file pointer when you open a file for binary sequential output and just start writing out data to it?

Do network file transport protocals like NetBEUI or TCP/IP specify a hard-coded size for a file pointer - perhaps something larger than 32 bits? If so, does Win-98 simply pad the extra bits with zero?

I wonder if a hook in the right place could watch the 32-bit file pointer during read/write operations, and when the pointer is reaching or has reached FF FF FF FF then set a flag to look to see if the next pointer value is 00 00 00 00 - which would indicate a roll-over, and simply append 01 to the 5'th byte of the pointer so that the pointer becomes 01 00 00 00 00, and keep checking for pointer monotonicity while the flag is set, and set the flag to 0 if monotonicity is broken for that particular file handle. This might only work for networked file operations where you know that the destination drive is running NTFS or exfat or fat64, but it might also work on a local FAT32 drive as long as correct cluster chaining takes place. It might not really matter that the FAT file-entry lists an incorrect file size so long as the actual clusters that are used are chained correctly and reported as being in-use.

Or maybe this is all messed up when DMA is used to read/write from a drive. But DMA would not be involved with network file access.

edit:

Just putting the issue of the size of the file pointer aside for a minute - what is really stopping FAT32 from storing files larger than 4 gb - assuming you can tolerate an incorrect file-size in the file table?

Why couldn't an NT-based OS (2k, XP, etc) using 8-byte file pointers, create files larger than 4 gb on a FAT32 volume? What does 2k or XP (etc) use for a file pointer anyways? Is it 8 bytes? 6 bytes?

Edited March 26, 2011 by wsxedcrfv

[dencorso]

... the 4 GiB (-1 byte) limit is intrinsic to the design of the FAT filesystem (no matter whether 12, 16 or 32, but only FAT-32 has volumes big enough to hit it), since it's due to representing the file size, in the directory structure, as an unsigned double-word (= 32 bits), resulting in a [(2^32)-1] max value, since the counting is from zero.

See also: FAT - Wikipedia

[Glenn9999]

Some interesting/complex things and I'll do my best to respond to some of them.

Well, I can confirm that winrar running on win-98, decompressing a multi-part RAR file that is larger than 4 gb to a network drive on a machine running NT4-server, will NOT display any error message during the decompression, and the decompression will appear to run fine all the way to completion, but the resulting file sitting on the NT4 server will be exactly 4.00 gb (4.2-something billion bytes).

Generally NTFS doesn't have the 4GB size limitation, but this has to do with what the OS supports. While NT4/NTFS can indeed support >4GB and has the drivers/libraries to support it properly (actually it's a kludge, but that's another topic), win98 wouldn't, so it would cause a problem. The server won't generate any errors because it doesn't know anything is up ("you copied me 4GB. Great!"), but the 98 file/network drivers don't know anything is up either (remember they were written at a time when a 4GB file was thought an absurdity).

What I don't understand - do you really need a file pointer when you open a file for binary sequential output and just start writing out data to it?

Yes. That's how files work and have so since the beginning of time. The point where you are reading/writing a file is denoted by a file pointer. It can't be eliminated.

Do network file transport protocals like NetBEUI or TCP/IP specify a hard-coded size for a file pointer - perhaps something larger than 32 bits? If so, does Win-98 simply pad the extra bits with zero?

Network protocols have nothing to do with file transfer. All they involve is passing data across a network connection and that is all the device knows. The network device knows nothing of file systems and vice versa. TCP/IP is a network protocol which just passes data. Now NetBEUI (NetBIOS) is a complete different thing - it allows organization and identification of the machines. In other words, it enables the machines to hook up and identify one another and identifies routing information. Now on top of that, there is file and printer sharing protocols, which is what you are confusing the network protocols with. The common one that works with your 98 is Samba. This is what has to bridge between the disk and data devices. All of this has been described in a really simplistic and not exactly accurate way, since there is much to study in networking theory regarding this topic (Google "OSI model" or "OSI 7 layer model" if you want to know more, I took a whole college class on this once). Point is, TCP/IP and NetBeui have nothing to do with file pointers.

I wonder if a hook in the right place could watch the 32-bit file pointer during read/write operations, and when the pointer is reaching or has reached FF FF FF FF then set a flag to look to see if the next pointer value is 00 00 00 00 - which would indicate a roll-over, and simply append 01 to the 5'th byte of the pointer so that the pointer becomes 01 00 00 00 00

Fifth byte makes it > 32-bit. 32 bits are 4 bytes. As was stated many times in your threads as well as the thread linked to, there is no way to change this for Windows 98 outside of rewriting the whole operating system in every place that it accesses files to use 64-bit pointers (the next step) as well as making a FAT64 file system for hard disks. You really can't kludge data storage. As you may guess, that is a huge task. While it could be done, that would require Microsoft open-sourcing 98 to even make it fruitful and that will never happen.

Just putting the issue of the size of the file pointer aside for a minute - what is really stopping FAT32 from storing files larger than 4 gb - assuming you can tolerate an incorrect file-size in the file table?

The file system itself does not allow storage of data amounts > 4GB. This goes into the way disks work - all the hardware knows is clusters and the file system is what makes sense of it for the OS. A file system really is a way to track what data are stored on disk and where. The problem is you can't "tolerate an incorrect file-size" in the allocation table, because it will result in data corruption and loss when the data are accessed. A file system is something you really don't want to kludge, for a number of reasons.

Why couldn't an NT-based OS (2k, XP, etc) using 8-byte file pointers, create files larger than 4 gb on a FAT32 volume? What does 2k or XP (etc) use for a file pointer anyways? Is it 8 bytes? 6 bytes?

Again because the file system doesn't support values greater than 4GB. As has been stated many times, Microsoft's answer has been NTFS so there won't be a FAT64 on their OSes. And they only created exFAT because they realized NTFS thrashes flash drives. You can't make anything work on something that is impossible to make work.

[rloew]

Winrar running on win-98 ought to be able to save it's decompression output to the (NTFS) networked drive, and VLC running on win-98 ought to be able to play large movies also from the networked drive.

Won't work. I tried that a long time ago. Windows 9x IOS uses 32-Bit File Pointers, so the Pointers are truncated before they even reach the FileSystem Drivers. This includes the Network Redirectors.

Well, I can confirm that winrar running on win-98, decompressing a multi-part RAR file that is larger than 4 gb to a network drive on a machine running NT4-server, will NOT display any error message during the decompression, and the decompression will appear to run fine all the way to completion, but the resulting file sitting on the NT4 server will be exactly 4.00 gb (4.2-something billion bytes). In this case, an mkv file that I could not get VLC to play. (I assume that NT4 does not have a 4 gb file-size limitation... ?)

What I don't understand - do you really need a file pointer when you open a file for binary sequential output and just start writing out data to it?

Do network file transport protocals like NetBEUI or TCP/IP specify a hard-coded size for a file pointer - perhaps something larger than 32 bits? If so, does Win-98 simply pad the extra bits with zero?

I wonder if a hook in the right place could watch the 32-bit file pointer during read/write operations, and when the pointer is reaching or has reached FF FF FF FF then set a flag to look to see if the next pointer value is 00 00 00 00 - which would indicate a roll-over, and simply append 01 to the 5'th byte of the pointer so that the pointer becomes 01 00 00 00 00, and keep checking for pointer monotonicity while the flag is set, and set the flag to 0 if monotonicity is broken for that particular file handle. This might only work for networked file operations where you know that the destination drive is running NTFS or exfat or fat64, but it might also work on a local FAT32 drive as long as correct cluster chaining takes place. It might not really matter that the FAT file-entry lists an incorrect file size so long as the actual clusters that are used are chained correctly and reported as being in-use.

Or maybe this is all messed up when DMA is used to read/write from a drive. But DMA would not be involved with network file access.

edit:

Just putting the issue of the size of the file pointer aside for a minute - what is really stopping FAT32 from storing files larger than 4 gb - assuming you can tolerate an incorrect file-size in the file table?

Why couldn't an NT-based OS (2k, XP, etc) using 8-byte file pointers, create files larger than 4 gb on a FAT32 volume? What does 2k or XP (etc) use for a file pointer anyways? Is it 8 bytes? 6 bytes?

There are a variety of Kludges that may provide some functionality, but nothing simple will make the functionality seamless.

The simplest would be to hack KERNEL32.DLL to create shadow files to hold the excess data and manage the File Pointers. The shadow files would create issues with file searches and file uniqueness but would allow programs to think they are working with larger files.

This would be a KernelEX type of Project.

Although the standard functions allow you to stream your reads and writes past the 4GB boundary without requiring a Pointer, internally all I/O must point to absolute positions in a file so errors will occur.

[Joseph_sw]

as in FAT32 i'm still curious about this one:

in directory entries, entry that has DIR attribute set, its filesize entry was 0, (that was i see using program such as norton diskedit)

so its theoreticaly possible to have a directory entry larger than 4GB provided its 'actual size' were multipe of cluster size.

but do win9x can actualy handle a very large directory entry which its actual cluster usages (x cluster' size) > 4GB ?

just think as a directory contains tons of files with extra long filenames, each.

Edited March 26, 2011 by Joseph_sw

[Guest wsxedcrfv]

Nobody has yet explained why FAT32 can't technically store files larger than 4gb. You all say that FAT32 codes the file-size as a 4-byte (32bit) value, and perhaps for operations like file-copy or move it's necessary to have exactly the right file-size, but I'm wondering why cluster-chaining can't technically grow beyond the 4gb file-size such that a file of any arbitrarily large size can be stored on fat32?

What other data structure within FAT32 other than the 4-byte file-size parameter would need to be modified to allow for files larger than 4 gb? Does cluster chaining have a limit at 4gb?

Microsoft limits something related to clusters to 28 bits, with the upper 4 bits being "reserved". What is all that about?

How about this: Why not imagine that the 32-bit value for the file-size is instead the number of clusters used by the file? Perhaps we set the high-bit of the file-size to 1 to indicate file-size in terms of clusters, and 0 to indicate file-size normally (in terms of bytes). That would be a good way to code large file storage within the current confines of FAT32. The work would be getting the OS to recognize the new convention.

[rloew]

Nobody has yet explained why FAT32 can't technically store files larger than 4gb. You all say that FAT32 codes the file-size as a 4-byte (32bit) value, and perhaps for operations like file-copy or move it's necessary to have exactly the right file-size, but I'm wondering why cluster-chaining can't technically grow beyond the 4gb file-size such that a file of any arbitrarily large size can be stored on fat32?

Many programs need to know the exact size of a File. Otherwise they are going to see extra data after the actual end of the file.

What other data structure within FAT32 other than the 4-byte file-size parameter would need to be modified to allow for files larger than 4 gb? Does cluster chaining have a limit at 4gb?

There is no inherent limit on chaining as far as I know.

Microsoft limits something related to clusters to 28 bits, with the upper 4 bits being "reserved". What is all that about?

I'm not sure why, but it does simplify some coding. Microsoft does use the reserved bits in the FAT Header.

How about this: Why not imagine that the 32-bit value for the file-size is instead the number of clusters used by the file? Perhaps we set the high-bit of the file-size to 1 to indicate file-size in terms of clusters, and 0 to indicate file-size normally (in terms of bytes). That would be a good way to code large file storage within the current confines of FAT32. The work would be getting the OS to recognize the new convention.

Not only would you still have the same problem as in the first part on large files, but you would limit normal operation to 2GiB. There are other fields in the Directory entry that would be more suitable.

@Joseph_sw: Windows 9x actually limits Directories to 2MiB (65536 entries). This is not related to Cluster chaining. "Actual" size is not an issue as unused entries in a Cluster must be either 0's or marked "Deleted".

Edited March 26, 2011 by rloew

[osRe]

The simplest solution and least risky one I see is to implement spanned-file functions using the same interface as the native one. Chuck these functions into a DLL with the same exports as KERNEL32.DLL, redirect everything else to the system DLL, and place this DLL in the directory of each app where the functionality is needed.

Not perfect integration, but should do, assuming the app uses 64-bit pointers internally.

[rloew]

The simplest solution and least risky one I see is to implement spanned-file functions using the same interface as the native one. Chuck these functions into a DLL with the same exports as KERNEL32.DLL, redirect everything else to the system DLL, and place this DLL in the directory of each app where the functionality is needed.

Not perfect integration, but should do, assuming the app uses 64-bit pointers internally.

Since you cannot have a local KERNEL32.DLL, you would need to modify your apps to use a different DLL.

[jds]

I don't have any BluRay hardware, but files/content > 4G are nothing new, they also exist on DVD.

In the case of DVD, the content is represented in chunks of just under 2G each, and you can copy and play these (*.VOB) files seamlessly with software such as VLC.

Can't the same be done with DVD? Even if (and I don't know) BluRay content is monolithic, surely one of those special copying programs should be able to transfer this content in a similar manner as for DVD? I realize this isn't a general solution, but if the main application for such huge files is BluRay content, this would mostly solve this > 4G requirement.

Perhaps adding support for > 4G (using similar file spanning techniques) to some key applications would meet the more general requirement, without the tricky task of hacking the O/S to add such support (and in the process, breaking some existing applications as a side-effect).

Joe.

[Guest wsxedcrfv]

I don't have any BluRay hardware, but files/content > 4G are nothing new, they also exist on DVD. In the case of DVD, the content is represented in chunks of just under 2G each, and you can copy and play these (*.VOB) files seamlessly with software such as VLC.

I'm not quite sure I know what you're talking about. All the VOB files I've ever seen on a DVD are exactly 1gb in size. I've never seen them larger than 1 gb, and certainly not more than 4 gb.

I think the answer is very simple: If your file is a movie file (and most likely it is) then convert it to MKV and then use winrar to break it up into smaller pieces (any size you want that is 4 gb or smaller) but tell winrar to use "storage mode" - no compression. VLC will play the file(s) no problem.