Jump to content

Exploring "SETUP /P I;S=detectbus" and Grub4Dos: 8½ times Win98se on USB-drive


deomsh

Recommended Posts


Good :), but it remains "queer".

Maybe (not that I can understand the code, mind you) the issue is not gzip, but rather with the later added lzma:

https://github.com/chenall/grub4dos/blob/master/stage2/gunzip.c

the header for gzip seem fine:

Quote

/* Little-Endian defines for the 2-byte magic number for gzip files */

#define GZIP_HDR_LE 0x8B1F

#define OLD_GZIP_HDR_LE 0x9E1F

the issue must be *somewhere* here:

https://github.com/chenall/grub4dos/blob/master/stage2/dec_lzma.c

It seems (to me) that the file format has not any fixed signature bytes:

https://svn.python.org/projects/external/xz-5.0.3/doc/lzma-file-format.txt

http://fileformats.archiveteam.org/wiki/LZMA_Alone

Quote

There is no signature, and no simple way to identify LZMA_Alone files just by looking at the header. That said, it is common for them to start with bytes 5D 00 00.

so it is a mis-detection of some kind.

I guess it should be reported as well as a bug[1].

For the moment I just posted a reference to this thread on reboot.pro a a "heads up":

http://reboot.pro/topic/22352-grub4dos-and-lnk-files-possible-lzma-collision/

jaclaz

[1] BTW personally I find totally absurd that such detection code runs seemingly on *any* file in several commands, it simply makes no sense given that gzip and LZMA files may represent what? 0.001% of files, the decision to ignore file extensions and only attempt decompression on (say) .gz and .lzma is IMHO flawed, but as often happens hypothetically changing that would break half of the batches and/or existing workflows, in the case of gzip it may be fine[2] (because of the header, though it is IMHO too short to be reliable) but with LZMA (headerless) the approach becomes an issue, and since LZMA files are much less common than gzip ones, limiting that to file with the .lzma (or *whatever*) extension could be more than acceptable.

[2] no, it isn't fine, more like "barely acceptable"

Edited by jaclaz
Link to comment
Share on other sites

14 hours ago, Wunderbar98 said:

Thank-you for the feedback and information Deomsh, sorry for the trouble.

@Wunderbar98

I am very happy with your contribution, I am looking now for a way to use two quotes only in case of spaces in PATH1/FILE1, instead in all individual parts containing spaces. In FATCOPY.G4B version 0.3 (forthcoming!) using only two spaces gives undesirable results: the whole root of DEVICE1 is copied, I am afraid with the /s-switch things will be even worse. I have to anticipate too how potential users (if any) will try to use the command-line.

If FATCOPY.G4B is ready, I will move on with FATMKDIR.G4B, FATDEL.G4B (both FAT-only) and FATLSDIR.G4B (for all file-systems supported by Grub4Dos). During my holiday this summer I made the first versions on my smartphone with Limbo Android/X86 (I had no PC around).

I am not sure if FATREN.G4B is really worthwhile, but the Grub4Dos-program FAT gives the interesting possibility to move a file to another directory. With my batchfile-approach this could be generalized to all files in directory, or groups of files using "*"-wildcards. So very difficult to resist such a project (for me ;)).

By the way, do you know a tool to make EXT3, EXT4 and ReiserFS partitions inside Virtual Box 6.0 on a VHD? So far I succeeded only with EXT2, and (unexpected) with Reiser2FS after I installed Suse 8.2 from an old DVD.

Edited by deomsh
Typo
Link to comment
Share on other sites

@jaclaz

Today I did some more tests to find out which bytes of header+GUID of a LNK-file gives trouble with Grub4DOS. During this experiment I came across lzma!

First I will describe the experiment, than I will give the print-screens. Prerequisite are 17 files in the range 4-20 bytes, I named them to their amount of bytes, with extension LNK. Most easy on a FAT-volume with fat mkfile.

With write I wrote (without raw!) the 'right' amount amount of bytes of the header+GUID of a LNK-file to each file, always one less. It appeared that the gzip-error popped up only during writing 20 and 19 bytes. From the 18 bytes file down to the 13 bytes file, no writing problem, but with debug on there was a warning: Grub4Dos switched to raw, because of detection of lzma-data which couldn't be decompressed. From the 12 bytes file down to the 4 bytes file NO warning.

Afterwards I'd try to read all files with cat --hex (without raw too): same problem with the 20 and 19 bytes files, like in my previous post about this subject. Same warning (with debug on) as during writing in case of the 18 bytes file down to the 13 bytes file. From the 12 bytes file down to the 4 bytes files no warning too (to fit on the print-screen I switched debug off earlier).

1832534916_VirtualBox_writedebugoffon20to17byte_20190909.png.0821ccc0fa5b4c2da3192d96d80a800b.png187895041_VirtualBox_writedebugon16to12byte_20190909.png.075b216248da3b509a8bee38d9297eb5.png988998324_VirtualBox_cat--hexdebugon20to16byte_20190909.png.047f891a93025fc5ac66831f75db48d5.png1784172649_VirtualBox_cat--hexdebugon15to4byte_20190909.png.98661813e2641a67596cf74220b3e358.png

12 hours ago, jaclaz said:

Maybe (not that I can understand the code, mind you) the issue is not gzip, but rather with the later added lzma:

https://github.com/chenall/grub4dos/blob/master/stage2/gunzip.c

the header for gzip seem fine:

Quote

/* Little-Endian defines for the 2-byte magic number for gzip files */

#define GZIP_HDR_LE 0x8B1F

#define OLD_GZIP_HDR_LE 0x9E1F

the issue must be *somewhere* here:

https://github.com/chenall/grub4dos/blob/master/stage2/dec_lzma.c

It seems (to me) that the file format has not any fixed signature bytes:

https://svn.python.org/projects/external/xz-5.0.3/doc/lzma-file-format.txt

http://fileformats.archiveteam.org/wiki/LZMA_Alone

Quote

There is no signature, and no simple way to identify LZMA_Alone files just by looking at the header. That said, it is common for them to start with bytes 5D 00 00.

so it is a mis-detection of some kind.

I will try to read your sources, but I (almost) never cared about file-headers, so I have not so much to say about this subject for now.

12 hours ago, jaclaz said:

I guess it should be reported as well as a bug[1].

For the moment I just posted a reference to this thread on reboot.pro a a "heads up":

http://reboot.pro/topic/22352-grub4dos-and-lnk-files-possible-lzma-collision/

I have red the post, I hope someone will respond - on your post on the vol-command & label still no comments (when I have finished FATMKDEV.G4B, a generalization of my earlier RD2HD-files, I will report in that thread if I found new issues).

Edited by deomsh
Corrections
Link to comment
Share on other sites

Hi Deomsh. Maybe your example from above is best as it gets the whole pathway from root in a single set of double quotes.
ls (hd2,0)/"lnkfiles/windows verkenner.lnk"

Don't know of a tool for EXT3, EXT4 and ReiserFS partitions inside Virtual Box 6.0. Personally still only use EXT2, even in modern GNU/Linux and with encrypted USB sticks. For me it's still the best with little overhead.

Link to comment
Share on other sites

  • 1 month later...
On 8/26/2020 at 9:51 PM, deomsh said:

@Wunderbar98

I am very happy with your contribution, I am looking now for a way to use two quotes only in case of spaces in PATH1/FILE1, instead in all individual parts containing spaces. In FATCOPY.G4B version 0.3 (forthcoming!) using only two spaces gives undesirable results: the whole root of DEVICE1 is copied, I am afraid with the /s-switch things will be even worse. I have to anticipate too how potential users (if any) will try to use the command-line.

I solved my problem using quotes & spaces on the FATCOPY.G4B command-line.

It appears that outside-quotes are removed if %~d1 / %~p1 / %~nx1 are used to read out DEVICE1 / PATH1 / FILE1 etcetera. So different from GRUB4DOS commands like ls, where a quote-sign before a DEVICE gives an error.

To avoid undesirable results because of remaing spaces in variables, all I had to do was an earlier call to my subroutine that changes normal spaces the the GRUB4DOS "\ "-spaces representation.

If quotes are used around PATH and /or FILE (or -superflous- even around DEVICE) things became more complicated if only one quote-sign is left after the readout (or three). It appears that a variable containing an odd number of quotes, regardless their position inside the variable, can not be written into memory with echo %var% > (md)0xMMMM+1. In this case the whole command-line is echo'd on screen, nothing happens further. I found a sort of workaround to add one quote-sign to the variable if this happens, but I am not fully satisfied, because the workaround in fact doesn't detect an odd number of quotes in the variable, only that writing a variable to (md) didn't happen.

Also quotes can't be detected by %var:~s,n% , although they are counted by echo %@retval% after set var=%var%

Any ideas?

Link to comment
Share on other sites

I don't understand.

Can you post some actual examples?

Normally (in "normal" batch) one would work "internally" without quotes and add them only once when needed in the actual "final" command .

Double quotes are however a large PITA in *any* script.

To write a string as is to a (md) you can use write, adding a string terminator, i.e.
 

set myvar="something""
write (md)0x3000+1 %myvar%\0
cat (md)0x3000+1

but then ?

jaclaz

 

 

 

Link to comment
Share on other sites

4 hours ago, jaclaz said:

I don't understand.

Can you post some actual examples?

My problem is to identify an odd number of quote-signs inside a variable, with the purpose to get rid of them with help of (md). Quotes are only used on the FATCOPY.G4B command-line in case of spaces in PATH and /or FILE. So only used in case of LFN's.

An example is in the picture below. Last five lines are normal copy-output, first three / four lines after fatcopy are written to screen for clarification.

1461183972_VirtualBox_MS-DOS7.1_evenodd_number_of_quote-signs_in_a_variable.png.d7015bd7d519454997c368e922882f27.png

If only two outside quotes are used, they are already removed by the %~1-readout of PATH (in the example DEVICE is root and FILE is not specified). But if each directory in PATH is surrounded by quotes, an odd number of quote-signs remains in the variable %sdir% after the readout. In this case the echo %sdir% > (md)0x3000+1 line is written to screen only, together with a part of my 'dirty' workaround:

if exist sdir && set /a succeed=0
# %mdmem% is normally 0x3000
if exist sdir && echo -n > (md)%mdmem%+1
if exist sdir && echo %sdir% > (md)%mdmem%+1 && set /a succeed=1
if exist sdir && if %succeed%==0 && set sdir=%sdir%"
if exist sdir && echo %sdir% > (md)%mdmem%+1 && set /a succeed=1
if exist sdir && if %succeed%==0 && echo && echo Use of qoutes is not valid && goto :eov
set quotes=
# " = 22 hex
if exist sdir && cat --locate=\x22 (md)%mdmem%+1 > nul
echo %@retval% | set quotes=
if exist sdir && if %quotes%>=1 && set quotes= && setlocal && set file=%sdir% && call :noquotes
if exist sdir && endlocal && set sdir=%file% && set file=
set quotes=

Once the variable containing quote-signs is written to (md)0x3000+1 I call following sub-routine to remove the quote-signs:

:noquotes
set seekbyt=
# " = 22hex
cat --locate=\x22 --number=1 (md)%mdmem%+1 | set seekbyt=
#%mdmem% is normally 0x3000
if not exist seekbyt && cat (md)%mdmem%+1 | set file= && goto :eof
set seekbyt=0x%seekbyt%
set /A seekbyt=%seekbyt%
set /A "skipbyt=%seekbyt% + 1"
dd if=(md)%mdmem%+1 of=(md)%mdmem%+1 bs=1 skip=%skipbyt% seek=%seekbyt%
goto :noquotes

Afterwards I use set sdir=%file% to get the source directory without quote-signs.

Third / fourth line in the picture above is after replacing all spaces in PATH by ls-spaces ("\ "), needed for proper operation. Everything is working as needed, only the echo of my dirty workaround bothers me, and because of its non-specific character.

Thanks for your solution to use write to write a variable to (md). :thumbup

I did some first tests and I think my problem is solved, but I have to rewrite some parts of my batch-file, because even set file=%sdir% && call :noquotes is not working if the variable contains an odd number of quote-signs. :no:

Edited by deomsh
Typo's
Link to comment
Share on other sites

Hi Deomsh. Not sure if this helps. In Unix-like environment an easy work-around is to declare variables containing spaces and double-quotes with a pair of single-quotes instead. It then doesn't matter how many spaces or double-quotes are embedded, provided there is only the single pair of single-quotes wrapping variable content.

a='space " quote " "'
echo $a
space " quote " "

Link to comment
Share on other sites

49 minutes ago, Wunderbar98 said:

In Unix-like environment an easy work-around is to declare variables containing spaces and double-quotes with a pair of single-quotes instead. It then doesn't matter how many spaces or double-quotes are embedded, provided there is only the single pair of single-quotes wrapping variable content.

@Wunderbar98 Thanks!

I just tested your Unix-suggestion, but doesn't seem to make any difference (see the print-screen below). Only the odd er even number of double-quotes seems to count in echo %a% > (md)0x3000+1. :dubbio:First part: variable with odd number of double-quotes inside single-quotes. Second part: variable with even number of double-quotes inside single-quotes

1468530393_VirtualBox_MS-DOS7.1_variable_with_odd_number_of_doublequote-signs_inside_single_quotes.png.a76d7e65d68a3c4ecdc3ce5a79813c0b.png

But at least I learned a single-quote is 27h :)

Jaclaz' solution to use write seems to work (see second print-screen). Only termination is different, without 0D 0A, but can be added if needed.

559873016_VirtualBox_MS-DOS7.1_variable_with_odd_number_of_doublequote-signs_no_single_quotes.png.78b6af7899812f2893b2a5c7008bf269.png

 

Link to comment
Share on other sites

This would also do (not that it is particularly elegant, actually it is awful to look at, but it seemingly works):

!BAT
set myvar="something" with space and two  spaces aND "" quotes"
echo %@retval% | set myvarl= 
set myvarnq=
set myvar
echo 
set /a counter=0 > nul
call :de_quote
set myvarWq="%myvarnq%"
set myvar
goto :eof

:de_quote
call set thischar=%^myvar:~%counter%,1%:
set /a counter=%counter%+1 > nul
#echo %thischar%
if not \%thischar%==\": set myvarnq=%myvarnq%%%thischar:~0,1%
if ":"=="%thischar:~0,1%" set myvarnq=%myvarnq:~0,-1% 
if %counter%==%myvarl% goto :eof
goto :de_quote

jaclaz

Edited by jaclaz
Link to comment
Share on other sites

@jaclaz Thanks a lot :thumbup

I am VERY happy to have a working example with call expanding %^var:~s,n% with a loop! I've red everything about it in Steve's documents I could find, but still didn't grasp the whole 'thing'.

In my understanding of your batch-file the double-quote character " can be evaluated with if %var%== if between two 'supported' characters.

I believe the colon you included in set thischar=%^myvar:%counter%,1%: is needed because of the spaces. But I found out the colon can be left outside if double-qoutes are used during setting thischar, then if not \%thischar%:==\" : is always working (and will the line if ":"=="%thischar:~0,1%" set myvarnq=%myvarnq:~0,-1% be unnecessary). To fit in one print-screen I changed 'something' in myvar too.

867398409_VirtualBox_MS-DOS7.1_de_quote.g4b_without_colon_in_var_thischar.png.e3362a4f55478780728839454f491572.png

Also substituting spaces with the needed ls-spaces ("\ ") is easygoing with your approach expanding variables with call.

But I think your idea to use write to get even a variable containing an odd number of double-quotes into (md),  together with my use of dd will be a bit faster. Only a fistful of spaces need to be cut out, instead rewriting the whole variable. Or am I wrong?

Link to comment
Share on other sites

Yep, it needs to be timed, but it is likely that the write to (md) works faster, though (with all due respect :)) you made it more complex (and slower) than needed with your dd loop (as it goes one char at the time).

Likely (but again it should be timed) something more like this (as well really "ugly") should be faster (less loops, hopefully only the needed ones):

!BAT
set /A mdmem=0x3000 > nul
set /A myskip=0x600000 > nul

set myvar="something" with space and two  spaces aND "" quotes"
echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0
cat (md)%mdmem%+1
echo
cat --locate=\x22 --replace=: (md)%mdmem%+1
set myvar=

call :Function13

set myvarWq="%myvar%"
set myvar
goto :eof

:Function13
call Fn.13 %myskip% ":" 
set /A myskip=%@retval% > nul
cat --skip=%myskip% (md)0+0x3001 | set mytoken=:
set /A myskip=%myskip%+%@retval% > nul
set myvar=%myvar%%%mytoken:~1%
call Fn.11 %myskip% ":" && goto :Function13
goto :eof

jaclaz

Link to comment
Share on other sites

7 hours ago, jaclaz said:

Likely (but again it should be timed) something more like this (as well really "ugly") should be faster (less loops, hopefully only the needed ones):

Interesting, but I will have to study the used Fn calls to understand more of it.

Your latest approach seems to work, but only if the variable begins and end with a double-quote character!

In my dd-approach this is not needed (and remaining double-quotes if used on the FATCOPY.G4B command-line will be not at begin or end of the variable containing the source path).

7 hours ago, jaclaz said:

Yep, it needs to be timed, but it is likely that the write to (md) works faster, though (with all due respect :)) you made it more complex (and slower) than needed with your dd loop (as it goes one char at the time).

Are there more precise timing possibities with GRUB4DOS? My timer in FATCOPY.G4B has a minimum of whole seconds, which is far to much for comparisons needed here. :thumbdown

Link to comment
Share on other sites

Well, only for the record, it wouldn't be that much a limitation to need to enclose the variable in quotes before calling the subroutine, but you are right :) that we can do "better".

This is more what I had in mind and needs not any exoteric function (and it is - I believe - actually readable/understandable).

!BAT
setlocal
set /A mdmem=0x3000 > nul
set /A myskip=0x600000 > nul

set myvar="something" with space and two  spaces aND "" quotes"
set myvarnq=
set myoffets=

echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0
cat --locate=\x22 (md)%mdmem%+1 | set myoffsets=
cat --locate=\x22 --replace=\x00 (md)%mdmem%+1
cat (md)%mdmem%+1 | set myvarnq=%myvarnq%

if not ""=="%myoffsets%" call :tokenize %myoffsets%
set myvar
goto :eof


:tokenize
set /A start=0x%1+1 > nul
cat --skip=%start% (md)%mdmem%+1 | set myvarnq=%myvarnq%
if not ""=="%2" shift && goto :tokenize
goto :eof 

About timing, it is like in batch, you need to put the routine in a loop (x100, x1000, x10000, whatever) and then the seconds are enough.

I would bet on this latter code, as there is no call to subroutine if there are no quotes and only as many calls as there are quotes in the variable.

Try it and see if it fits the bill.

jaclaz

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...