Exploring "SETUP /P I;S=detectbus" and Grub4Dos: 8½ times Win98se on USB-drive

[deomsh]

On 11/10/2020 at 1:33 PM, jaclaz said:

Maybe you could use (like in other grub4dos commands) the hyphen as separator for switches, i.e. -b, -s, -q, etc., otherwise something *loosely like* :

set param1=%1
if not "%param1:~-1,1%==/" <do something inserting a / as first parameter>

might do. this should "catch" lines where you omit the / (because in this case the second parameter (switch) is shifted to first) while keeping untouched the provided one, either a single / or a path (like (hd3,0)/) terminated with /.

Thanks for all your ideas.

I worked on the problem. As far as I can see, I can maintain my beloved /-switches with a few modifications and a sub-routine to split up %@root% in DEVICE and /PATH.

I believe the only file-representations that are NOT possible anymore are filenames same as switches AND on the root, represented by '/'. But without preceding '/' there seems to be no problems. See the first print-screen.

Even file-names starting with '-' are possible (unless '+', '-' is not a forbidden character in MS-DOS).

Also my problems with '/s' are solved, The output of the '/s /b'-switches combination of DIR in MS-DOS surprised me (can't remember I've used it before), so I added this combination to FATLSDIR.G4B. Even empty directories are shown in the short DIR-way, (fd0)/FATLSDIR/SUBDIR1 is the example. See second print-screen.

If DEVICE is used too, everything is as before. See last print-screen. Watch the sort order: directories first in both cases, empty directories are given one time only with '/s /b'.

Edited November 14, 2020 by deomsh
Typo, correction

[deomsh]

While (still) testing FATCOPY.G4B version 3, I found a strange result after copying 36 000 files to a directory (testing in VBOX 6). FATCOPY counted the right number of files / copy-size. After I mounted the VHD Windows reported happily the same result (and even the type of files I used, not the filename and the method of numbering of course!).

See first print-screen:

MS-DOS saw only 32 767 files with DIR /W, in the same directory (see second print-screen):

To be sure it was not related to the /W-switch, I checked the source directory I used without /W (see third print-screen):

I can't find anything about this number in combination with DIR. Is this a known limitation of DIR in MS-DOS 7.1, or otherwise?

BTW: SCANDISK.EXE found no errors on both (FAT32)-drives after copying, so FAT did a nice job. Thank you Chenall.

Edited December 10, 2020 by deomsh
Typo

[jaclaz]

Very likely .

32767 in hex is 0x7FFF.

So it is a limit for a (signed) integer in 2 bytes.

The actual limit for the number of files in FAT32 is anyway 65535 (or 0xFFFF), i.e. 2 bytes unsigned integer;
https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory

The reference document (by Microsoft) as often happens says this (and the contrary of it) explicitly:

http://download.microsoft.com/download/0/8/4/084c452b-b772-4fe5-89bb-a0cbf082286a/fatgen103.doc

(at the very end, pages 33/34)

BUT see also:

http://reboot.pro/topic/19643-winsxs-hardlinked-files/

https://stackoverflow.com/questions/37135262/fat32-number-of-files-per-directory-limit

there is the added complication of file name length (it shouldn't affect specifically your case since they are 8.3 "normal" file names)

Anyway, I wouldn't be surprised if in MS-DOS 7.1 the DIR used a signed integer instead of an unsigned one.

jaclaz

[deomsh]

Thanks a lot, nice info.

I did a large Google-search on '0x7FFF' and 'MS-DOS', but no results.

I also checked MS-DOS 7.1 in Real mode on another machine, same limit of DIR: 32 767 files. But Windows 98SE seems not to be affected, both Explorer and DIR in a MS-DOS Windows gave the right number of files (in the meantime 40 000 ). See the print-screens.

But on the other hand: Total Commander (v 9.22a) complained there where 'too many files' after opening the directory above, and showed afterwards 32 699 files only.

[jaclaz]

Most probably the "issue" has been "fixed" in Win9x but not "backported" to real mode MS-DOS 7.x.

jaclaz

 

[deomsh]

I did some more tests in Real mode. 

First I tried 4DOS as command-interpreter, but 4DOS gave with a much lower number of files in a directory already an 'out of memory' error.

Second I used a Freedos bootdisk in VBOX 6. Freedos had no problems with more than 32 767 files in a directory (see print-screen).

So at least I can check my test-outcomes in Real mode inside VBOX 6. 

[deomsh]

FATTOOLS: FATCOPY.G4B VERSION 0.3

I worked for a long time on version 3, which resulted in many changes and new features too. Sadly the size of the script more than doubled.  Download is at the end of this post.

Updates:

• Improved switch /d to add source-directory to a given target-directory instead of to the root only.
• Changing of base memory-address with optional third argument on the FATCOPY.G4B command-line instead by switch. Addresses 0x0000-0x2999 and 0x4000-0xD460 will be refused.
• The switch /ls is auto-set if a related switch is used, /ls is only needed if switching to 'ls' directory-parsing on a FAT-source.
• Copy from NTFS: ‘$’-directories/files are copied only with the switch /$, ment for (self-declared) experts only.
• Separation of empty directories / zero-byte files not problematic anymore, except for NTFS-source (switch /e for 'always seen as directory' needed on NTFS only).
• Improved error-detection on the FATCOPY.G4B command-line.
• Improved verify-function with crc32 instead of file-size.
• Right detection of file-size if copying 2GB-4GB files (with cat --length=0 FILE, %@retval% is limited to 0x7FFFFFFF).
• After detecting the right free space ('fat' is buggy on FAT32-partitions above 4GB), 'fatmini' is used as workaround (mandatory as soon a FAT32-target > 4GB is detected).
• All file-investigating functions now 'raw' (for right handling of LNK-files - thanks to Jaclaz).
• Earlier added visibility of omitted directories/files will be undone if using a '*'-wildcard (with '*.*' too).
• Less error statistics to reduce number of variables (number of errors during making directories maintained). Reduced statistics after finishing copying a directory (in case of switch /s).

New features:

• Copy from EXT2/3/4 added, my earlier problems seems to be related to the Windows EXT2-driver, with real EXT2/3/4-partitions there are no problems anymore. The ‘lost+found’ directory is only seen/copied with the switch /l+f.
• Copy from EXFAT, ReIsErFs and ReIsEr2Fs added (ReIsErFs untested).
• Copy from UDF ('ls' directory-parsing seems to be limited to Mode 1 UDF 2.01).
• Copying-mode switches to 'case-sensitive' if a case-sensitive File System is detected. Can be undone with switch /nocase (target is FAT only, so not case-sensitive. 8+3 file-names identical except for case, can not both be copied without renaming before).
• Separate copy engine using (raw) 'dd' for files that can't be copied by 'fat' ('fat mkfile' is still needed to make empty target-files with same size).
• Copy files with File Number Suffix, like Joliet. One file-number only [not]copied possible with switch /filenum:[-]number (1-32767).
• Incremental copying, uses crc32. Overwriting without notice if source-file has been changed (switch /i).
• Ignore chosen extensions during copying with switch /noext:ext (number limited - in theory - by size of one variable, so about 125 extensions, including semicolon-separator).
• Copying of Long File Names (LFN) to their Short File Name (SFN) counterpart (switch /lfn). SFN is NOT red from source File System, but generated according to Microsoft MS-DOS/Win9x specification. Highest number shown AFTER tilde in case of files with identical SFN's: 65535 in one directory (no generation of NTFS-method of numbering). Char '!' and space added to the list of forbidden characters. Wildcard '*' can be used AFTER max 16 chars of the name-part and AFTER max 6 chars of the extension-part of a LFN. Source-length supported: 511 chars (target still 255 chars).
• Saving of LFN's if switch /lfn:@ is used. Original LFN of  'SFN~num.ext' is saved as text to 'SFN@num.ext' in case of files and to 'SFN#num.ext' in case of directories (maximum 255 chars, maximum 32767 'SFN~num.ext'-files and files containing a LFN both, in one directory). Overwritten if LFN on source is changed and if no new 'SFN~num.ext' is made (source- and target-LFN are shown before dialog). Automatic generation of new 'SFN~num.ext' if needed, with switch /i. Existing 'SFN@num.ext' and 'SFN#num.ext' auto-saved if the corresponding 'SFN~num.ext'-file is copied, or 'SFN~num.ext'-directory is made (function can be undone with /x:@ in case of files, or with /x:-@ altogether). Includes statistics related to number of 'saved' LFN's, after finish copying.
• On the FATCOPY.G4B command-line both source and target accepts LFN's, if containing spaces or '=' always between "double quotes" (first one also possible before DEVICE). Quote-handling uses 'write' to overcome problems if an odd number of quotes remains after parsing of the command-line (thanks to Jaclaz!). Escaped 'ls'-spaces ('\ ') not shown in output during copying ('write' unexpected removes the escape-char when writing a variable with '\ ' and leaves the space).
• Fully compatible to copying from a root containing spaces, earlier set with 'ls'-spaces (no quotes if using Grub4Dos directly!), and source or target starts with '/' - representing the actual root ('ls' can't!). Full path to FATCOPY.G4B must be given, or set before.
• With switch /lfn compatible to at least Mode 1 Joliet level 1/level X and ISO9660 level 2/level X.

Some highlights

To make this post not too long,  only a few print-screens of the capabilities of FATCOPY.G4B v0.3 will be given. Command-line is showed at the bottom, if applicable (‘pager on’ doesn’t show commands).

First print-screen shows the verify-switch in action during copying my original Windows 98 SE CD. ‘Verify’ tries three times before giving up (the showed first two files where the only one not copied, third file was copied). Only needed on very slow or ‘bad’ media.

Second print-screen shows copying of an ISO9660-Joliet DVD, semicolon and File Suffix Number are simply cut-off before ‘dd’ is used to copy them. Only on a FAT-source ‘fat’ will read-out the corresponding SFN, but ‘ls’ can’t. Besides: now same result as MS-DOS for SFN’s. LFN's can be copied too with switch /lfn[:@] (no switches used here).

Third print-screen shows the switch /lfn in action. But if the first six chars of SFN are equal, this is more or less a blind process, although the dialog gives four options.

Fourth and fifth print-screen shows the ‘:@’-addition to the switch /lfn in action, and give my solution to the so called ‘Micros~1/Micros~2’ problem, and with switch /i fully automatic. Incremental copying if a new source file exists can be seen on print-screen five.

Sixth print-screen shows preservation of earlier saved LFN’s if tilded’ SFN’s are copied.

Seventh print-screen shows the last output-lines after copying a full drive with use of /lfn:@ and /noext:ext to exclude some file-types (in this case together at least 1,2GB). The final result is showed in print-screen eight.     

Copying is still maximum 21-23 directories deep, but this limit seems (partly?) dependent too on length of names involved (the error that will come up with next directory-level is: ‘FAULT: <<<<<<<<<<SYSTETM STATCK RUNOUT>>>>>>>>>’ and is generated by GRUB.EXE, not by FAT). Last print-screen shows inspection with (still forthcoming) FATLSDIR.G4B after copying LFN’s with originally 255 chars, and same error with depth of 2+21 directories.   

Remaining problems

Read-only files on the target will not be overwritten or deleted by ‘fat’ (by ‘fatmini’ neither). ‘fat’ directory-parsing allows not-copying files with certain attributes, but ‘ls’ directory-parsing can’t.

Since there seems to be no such a program like ATTRIB.EXE for Grub4Dos, in this case the overwrite-dialog has to be used for each file to be sure. Luckily most read-only files resides on the root of a FAT partition. So after processing the root, the overwrite dialog gives each time the options ‘A’ (overwrite all) and ‘S’ (overwrite never) too, and one can be chosen if desired.

Download FATCOPY3.ZIP
The zip contains a SFV-file with the crc32-value of FATCOPY.G4B too.

FATCOPY3.ZIP

Edited December 28, 2020 by deomsh
Corrections

[jaclaz]

Very good .

As a side-side note, since you mentioned .iso's, no idea if it applies to your nice batch, but doing a dashun9 test  is IMHO not that bad an idea:

https://msfn.org/board/topic/182116-winsetupfromusb-problem-installing-xp-on-legacy-system/?tab=comments#comment-1191406

jaclaz

 

[deomsh]

7 hours ago, jaclaz said:

As a side-side note, since you mentioned .iso's, no idea if it applies to your nice batch, but doing a dashun9 test  is IMHO not that bad an idea:

Thanks, I will try

So far I relied heavily on Imgburn for ISO's and on my CD/DVD-collection. I had a bad time testing ISO's because I tried a new version of Grub4Dos too, which gave problems with UDF. Maybe it's good to run some new tests. I saw that in between 'mkisofs' evolved to CDRTOOLS-3.02a10-win-x86_64 (following an old link in your post). Luckily I possess Windows 10 64-bit. I hope the command-line will accept your script.

My script changes nothing on the source-side, I rely on what 'ls' is able to read, and on 'fat' or 'dd' for copying. I started my script to be able to copy a full directory, it only evolved a little bit.

Case comes only into play while writing a source-directory and/or source-file and with my use of wildcards. So far I found real case-sensitivity only necessary on Linux File Systems, although as soon UDF is detected my script set case-sensitivity too. Personally I'm not so fond of case-sensitivity, and my switch /nocase can be used in most cases.

On the first print-screen is one of my tests to cope with case-sensitivity on ReIsEr2Fs (I see I forgot a '/', but due my new processing of source and target -this time- without consequences).

The switch /nocase can be seen in action in the next print-screen. Unless a given file is copied, the case of the output shown is identical to the redirected output of 'ls' (or should be, unless I made a mistake). The target is a different thing of course, 'fat' will deliver uppercase only (in this case not identical to the shown output - I am not sure if I changed this in my final version 0.3, but it's not a serious matter).

[deomsh]

During the final tests of FATLSDIR.G4B I found NTFS to be very slow. Especially a problem if parsing directories with many files, or if searching for a file or a directory. After some changes I couldn't get any faster. See following comparisons made with LOOPTEST.G4B (working version, unpublished).

Performance of FAT16 is best, FAT32 'good' too. EXFAT already much slower (different filesize!), but NTFS is really bad.

I think the bottleneck is the speed of raw cat --length=0 FILE - indispensable to identify directories and used to get file-size too.

With help of LOOPTEST.G4B is tried to reproduce the behavior if this internal Grub4Dos command. The story is a bit complicated, because LOOPTEST.G4B has one command-line only, so all variables are set to one fixed value.

Following print-screen shows my approach, using some magic (and some patience to get a 'working' command-line. Watch use of '#' instead of %^% !

• First command is FATLSDIR.G4B, showing first 5 files to test with cat. By chance I had numbered files, used in earlier tests.
• After deleting all variables, the third command is LOOPTEST.G4B, with ONE Grub4Dos command-line to execute (too long to fit on the screen).
• Fourth command shows the variables 'fed' during the looptest to Grub4Dos for execution. Also final values of the three variables, changed after each single test-loop.

The time of 1000 executions of the Grub4Dos command-line is almost the same as the difference between FAT and NTFS (see second print-screen above).

I tested the difference between cat and raw cat too, without conclusive results. 'Raw' seems a little bit faster, but access-time to a file in VBOX 6 has a considerable variation, about 6.5%.

To me it seems that the other commands used, can't be of much influence on speed. See last print screen (showing 'middle-part' of the command-lines too).

BTW after last command value of 'size' is no longer file-size.

So my question is if the slow file-access in case of NTFS is by default, or if there are faster solutions with Grub4Dos to get the size of a file?

Edit

I did some more test 'in the wild' on a 500GB sata HD/SSD hybrid disk. This time there where sometimes long looptests with FAT32/FAT16 too, so I can not rule out hardware /dataset related issues. For now I will skip these experiments.

Edited February 16, 2021 by deomsh
Addition