Wunderbar98 Posted August 26, 2020 Share Posted August 26, 2020 Thank-you for the feedback and information Deomsh, sorry for the trouble. Link to comment Share on other sites More sharing options...
jaclaz Posted August 26, 2020 Share Posted August 26, 2020 (edited) Good , but it remains "queer". Maybe (not that I can understand the code, mind you) the issue is not gzip, but rather with the later added lzma: https://github.com/chenall/grub4dos/blob/master/stage2/gunzip.c the header for gzip seem fine: Quote /* Little-Endian defines for the 2-byte magic number for gzip files */ #define GZIP_HDR_LE 0x8B1F #define OLD_GZIP_HDR_LE 0x9E1F the issue must be *somewhere* here: https://github.com/chenall/grub4dos/blob/master/stage2/dec_lzma.c It seems (to me) that the file format has not any fixed signature bytes: https://svn.python.org/projects/external/xz-5.0.3/doc/lzma-file-format.txt http://fileformats.archiveteam.org/wiki/LZMA_Alone Quote There is no signature, and no simple way to identify LZMA_Alone files just by looking at the header. That said, it is common for them to start with bytes 5D 00 00. so it is a mis-detection of some kind. I guess it should be reported as well as a bug[1]. For the moment I just posted a reference to this thread on reboot.pro a a "heads up": http://reboot.pro/topic/22352-grub4dos-and-lnk-files-possible-lzma-collision/ jaclaz [1] BTW personally I find totally absurd that such detection code runs seemingly on *any* file in several commands, it simply makes no sense given that gzip and LZMA files may represent what? 0.001% of files, the decision to ignore file extensions and only attempt decompression on (say) .gz and .lzma is IMHO flawed, but as often happens hypothetically changing that would break half of the batches and/or existing workflows, in the case of gzip it may be fine[2] (because of the header, though it is IMHO too short to be reliable) but with LZMA (headerless) the approach becomes an issue, and since LZMA files are much less common than gzip ones, limiting that to file with the .lzma (or *whatever*) extension could be more than acceptable. [2] no, it isn't fine, more like "barely acceptable" Edited August 26, 2020 by jaclaz Link to comment Share on other sites More sharing options...
deomsh Posted August 26, 2020 Author Share Posted August 26, 2020 (edited) 14 hours ago, Wunderbar98 said: Thank-you for the feedback and information Deomsh, sorry for the trouble. @Wunderbar98 I am very happy with your contribution, I am looking now for a way to use two quotes only in case of spaces in PATH1/FILE1, instead in all individual parts containing spaces. In FATCOPY.G4B version 0.3 (forthcoming!) using only two spaces gives undesirable results: the whole root of DEVICE1 is copied, I am afraid with the /s-switch things will be even worse. I have to anticipate too how potential users (if any) will try to use the command-line. If FATCOPY.G4B is ready, I will move on with FATMKDIR.G4B, FATDEL.G4B (both FAT-only) and FATLSDIR.G4B (for all file-systems supported by Grub4Dos). During my holiday this summer I made the first versions on my smartphone with Limbo Android/X86 (I had no PC around). I am not sure if FATREN.G4B is really worthwhile, but the Grub4Dos-program FAT gives the interesting possibility to move a file to another directory. With my batchfile-approach this could be generalized to all files in directory, or groups of files using "*"-wildcards. So very difficult to resist such a project (for me ). By the way, do you know a tool to make EXT3, EXT4 and ReiserFS partitions inside Virtual Box 6.0 on a VHD? So far I succeeded only with EXT2, and (unexpected) with Reiser2FS after I installed Suse 8.2 from an old DVD. Edited August 26, 2020 by deomsh Typo Link to comment Share on other sites More sharing options...
deomsh Posted August 26, 2020 Author Share Posted August 26, 2020 (edited) @jaclaz Today I did some more tests to find out which bytes of header+GUID of a LNK-file gives trouble with Grub4DOS. During this experiment I came across lzma! First I will describe the experiment, than I will give the print-screens. Prerequisite are 17 files in the range 4-20 bytes, I named them to their amount of bytes, with extension LNK. Most easy on a FAT-volume with fat mkfile. With write I wrote (without raw!) the 'right' amount amount of bytes of the header+GUID of a LNK-file to each file, always one less. It appeared that the gzip-error popped up only during writing 20 and 19 bytes. From the 18 bytes file down to the 13 bytes file, no writing problem, but with debug on there was a warning: Grub4Dos switched to raw, because of detection of lzma-data which couldn't be decompressed. From the 12 bytes file down to the 4 bytes file NO warning. Afterwards I'd try to read all files with cat --hex (without raw too): same problem with the 20 and 19 bytes files, like in my previous post about this subject. Same warning (with debug on) as during writing in case of the 18 bytes file down to the 13 bytes file. From the 12 bytes file down to the 4 bytes files no warning too (to fit on the print-screen I switched debug off earlier). 12 hours ago, jaclaz said: Maybe (not that I can understand the code, mind you) the issue is not gzip, but rather with the later added lzma: https://github.com/chenall/grub4dos/blob/master/stage2/gunzip.c the header for gzip seem fine: Quote /* Little-Endian defines for the 2-byte magic number for gzip files */ #define GZIP_HDR_LE 0x8B1F #define OLD_GZIP_HDR_LE 0x9E1F the issue must be *somewhere* here: https://github.com/chenall/grub4dos/blob/master/stage2/dec_lzma.c It seems (to me) that the file format has not any fixed signature bytes: https://svn.python.org/projects/external/xz-5.0.3/doc/lzma-file-format.txt http://fileformats.archiveteam.org/wiki/LZMA_Alone Quote There is no signature, and no simple way to identify LZMA_Alone files just by looking at the header. That said, it is common for them to start with bytes 5D 00 00. so it is a mis-detection of some kind. I will try to read your sources, but I (almost) never cared about file-headers, so I have not so much to say about this subject for now. 12 hours ago, jaclaz said: I guess it should be reported as well as a bug[1]. For the moment I just posted a reference to this thread on reboot.pro a a "heads up": http://reboot.pro/topic/22352-grub4dos-and-lnk-files-possible-lzma-collision/ I have red the post, I hope someone will respond - on your post on the vol-command & label still no comments (when I have finished FATMKDEV.G4B, a generalization of my earlier RD2HD-files, I will report in that thread if I found new issues). Edited August 26, 2020 by deomsh Corrections Link to comment Share on other sites More sharing options...
Wunderbar98 Posted August 27, 2020 Share Posted August 27, 2020 Hi Deomsh. Maybe your example from above is best as it gets the whole pathway from root in a single set of double quotes. ls (hd2,0)/"lnkfiles/windows verkenner.lnk" Don't know of a tool for EXT3, EXT4 and ReiserFS partitions inside Virtual Box 6.0. Personally still only use EXT2, even in modern GNU/Linux and with encrypted USB sticks. For me it's still the best with little overhead. Link to comment Share on other sites More sharing options...
deomsh Posted October 2, 2020 Author Share Posted October 2, 2020 On 8/26/2020 at 9:51 PM, deomsh said: @Wunderbar98 I am very happy with your contribution, I am looking now for a way to use two quotes only in case of spaces in PATH1/FILE1, instead in all individual parts containing spaces. In FATCOPY.G4B version 0.3 (forthcoming!) using only two spaces gives undesirable results: the whole root of DEVICE1 is copied, I am afraid with the /s-switch things will be even worse. I have to anticipate too how potential users (if any) will try to use the command-line. I solved my problem using quotes & spaces on the FATCOPY.G4B command-line. It appears that outside-quotes are removed if %~d1 / %~p1 / %~nx1 are used to read out DEVICE1 / PATH1 / FILE1 etcetera. So different from GRUB4DOS commands like ls, where a quote-sign before a DEVICE gives an error. To avoid undesirable results because of remaing spaces in variables, all I had to do was an earlier call to my subroutine that changes normal spaces the the GRUB4DOS "\ "-spaces representation. If quotes are used around PATH and /or FILE (or -superflous- even around DEVICE) things became more complicated if only one quote-sign is left after the readout (or three). It appears that a variable containing an odd number of quotes, regardless their position inside the variable, can not be written into memory with echo %var% > (md)0xMMMM+1. In this case the whole command-line is echo'd on screen, nothing happens further. I found a sort of workaround to add one quote-sign to the variable if this happens, but I am not fully satisfied, because the workaround in fact doesn't detect an odd number of quotes in the variable, only that writing a variable to (md) didn't happen. Also quotes can't be detected by %var:~s,n% , although they are counted by echo %@retval% after set var=%var% Any ideas? Link to comment Share on other sites More sharing options...
jaclaz Posted October 2, 2020 Share Posted October 2, 2020 I don't understand. Can you post some actual examples? Normally (in "normal" batch) one would work "internally" without quotes and add them only once when needed in the actual "final" command . Double quotes are however a large PITA in *any* script. To write a string as is to a (md) you can use write, adding a string terminator, i.e. set myvar="something"" write (md)0x3000+1 %myvar%\0 cat (md)0x3000+1 but then ? jaclaz Link to comment Share on other sites More sharing options...
deomsh Posted October 2, 2020 Author Share Posted October 2, 2020 (edited) 4 hours ago, jaclaz said: I don't understand. Can you post some actual examples? My problem is to identify an odd number of quote-signs inside a variable, with the purpose to get rid of them with help of (md). Quotes are only used on the FATCOPY.G4B command-line in case of spaces in PATH and /or FILE. So only used in case of LFN's. An example is in the picture below. Last five lines are normal copy-output, first three / four lines after fatcopy are written to screen for clarification. If only two outside quotes are used, they are already removed by the %~1-readout of PATH (in the example DEVICE is root and FILE is not specified). But if each directory in PATH is surrounded by quotes, an odd number of quote-signs remains in the variable %sdir% after the readout. In this case the echo %sdir% > (md)0x3000+1 line is written to screen only, together with a part of my 'dirty' workaround: if exist sdir && set /a succeed=0 # %mdmem% is normally 0x3000 if exist sdir && echo -n > (md)%mdmem%+1 if exist sdir && echo %sdir% > (md)%mdmem%+1 && set /a succeed=1 if exist sdir && if %succeed%==0 && set sdir=%sdir%" if exist sdir && echo %sdir% > (md)%mdmem%+1 && set /a succeed=1 if exist sdir && if %succeed%==0 && echo && echo Use of qoutes is not valid && goto :eov set quotes= # " = 22 hex if exist sdir && cat --locate=\x22 (md)%mdmem%+1 > nul echo %@retval% | set quotes= if exist sdir && if %quotes%>=1 && set quotes= && setlocal && set file=%sdir% && call :noquotes if exist sdir && endlocal && set sdir=%file% && set file= set quotes= Once the variable containing quote-signs is written to (md)0x3000+1 I call following sub-routine to remove the quote-signs: :noquotes set seekbyt= # " = 22hex cat --locate=\x22 --number=1 (md)%mdmem%+1 | set seekbyt= #%mdmem% is normally 0x3000 if not exist seekbyt && cat (md)%mdmem%+1 | set file= && goto :eof set seekbyt=0x%seekbyt% set /A seekbyt=%seekbyt% set /A "skipbyt=%seekbyt% + 1" dd if=(md)%mdmem%+1 of=(md)%mdmem%+1 bs=1 skip=%skipbyt% seek=%seekbyt% goto :noquotes Afterwards I use set sdir=%file% to get the source directory without quote-signs. Third / fourth line in the picture above is after replacing all spaces in PATH by ls-spaces ("\ "), needed for proper operation. Everything is working as needed, only the echo of my dirty workaround bothers me, and because of its non-specific character. Thanks for your solution to use write to write a variable to (md). I did some first tests and I think my problem is solved, but I have to rewrite some parts of my batch-file, because even set file=%sdir% && call :noquotes is not working if the variable contains an odd number of quote-signs. Edited October 2, 2020 by deomsh Typo's Link to comment Share on other sites More sharing options...
Wunderbar98 Posted October 2, 2020 Share Posted October 2, 2020 Hi Deomsh. Not sure if this helps. In Unix-like environment an easy work-around is to declare variables containing spaces and double-quotes with a pair of single-quotes instead. It then doesn't matter how many spaces or double-quotes are embedded, provided there is only the single pair of single-quotes wrapping variable content. a='space " quote " "' echo $a space " quote " " Link to comment Share on other sites More sharing options...
deomsh Posted October 2, 2020 Author Share Posted October 2, 2020 49 minutes ago, Wunderbar98 said: In Unix-like environment an easy work-around is to declare variables containing spaces and double-quotes with a pair of single-quotes instead. It then doesn't matter how many spaces or double-quotes are embedded, provided there is only the single pair of single-quotes wrapping variable content. @Wunderbar98 Thanks! I just tested your Unix-suggestion, but doesn't seem to make any difference (see the print-screen below). Only the odd er even number of double-quotes seems to count in echo %a% > (md)0x3000+1. First part: variable with odd number of double-quotes inside single-quotes. Second part: variable with even number of double-quotes inside single-quotes But at least I learned a single-quote is 27h Jaclaz' solution to use write seems to work (see second print-screen). Only termination is different, without 0D 0A, but can be added if needed. Link to comment Share on other sites More sharing options...
jaclaz Posted October 3, 2020 Share Posted October 3, 2020 (edited) This would also do (not that it is particularly elegant, actually it is awful to look at, but it seemingly works): !BAT set myvar="something" with space and two spaces aND "" quotes" echo %@retval% | set myvarl= set myvarnq= set myvar echo set /a counter=0 > nul call :de_quote set myvarWq="%myvarnq%" set myvar goto :eof :de_quote call set thischar=%^myvar:~%counter%,1%: set /a counter=%counter%+1 > nul #echo %thischar% if not \%thischar%==\": set myvarnq=%myvarnq%%%thischar:~0,1% if ":"=="%thischar:~0,1%" set myvarnq=%myvarnq:~0,-1% if %counter%==%myvarl% goto :eof goto :de_quote jaclaz Edited October 3, 2020 by jaclaz Link to comment Share on other sites More sharing options...
deomsh Posted October 3, 2020 Author Share Posted October 3, 2020 @jaclaz Thanks a lot I am VERY happy to have a working example with call expanding %^var:~s,n% with a loop! I've red everything about it in Steve's documents I could find, but still didn't grasp the whole 'thing'. In my understanding of your batch-file the double-quote character " can be evaluated with if %var%== if between two 'supported' characters. I believe the colon you included in set thischar=%^myvar:%counter%,1%: is needed because of the spaces. But I found out the colon can be left outside if double-qoutes are used during setting thischar, then if not \%thischar%:==\" : is always working (and will the line if ":"=="%thischar:~0,1%" set myvarnq=%myvarnq:~0,-1% be unnecessary). To fit in one print-screen I changed 'something' in myvar too. Also substituting spaces with the needed ls-spaces ("\ ") is easygoing with your approach expanding variables with call. But I think your idea to use write to get even a variable containing an odd number of double-quotes into (md), together with my use of dd will be a bit faster. Only a fistful of spaces need to be cut out, instead rewriting the whole variable. Or am I wrong? Link to comment Share on other sites More sharing options...
jaclaz Posted October 3, 2020 Share Posted October 3, 2020 Yep, it needs to be timed, but it is likely that the write to (md) works faster, though (with all due respect ) you made it more complex (and slower) than needed with your dd loop (as it goes one char at the time). Likely (but again it should be timed) something more like this (as well really "ugly") should be faster (less loops, hopefully only the needed ones): !BAT set /A mdmem=0x3000 > nul set /A myskip=0x600000 > nul set myvar="something" with space and two spaces aND "" quotes" echo -n > (md)%mdmem%+1 write (md)%mdmem%+1 %myvar%\0 cat (md)%mdmem%+1 echo cat --locate=\x22 --replace=: (md)%mdmem%+1 set myvar= call :Function13 set myvarWq="%myvar%" set myvar goto :eof :Function13 call Fn.13 %myskip% ":" set /A myskip=%@retval% > nul cat --skip=%myskip% (md)0+0x3001 | set mytoken=: set /A myskip=%myskip%+%@retval% > nul set myvar=%myvar%%%mytoken:~1% call Fn.11 %myskip% ":" && goto :Function13 goto :eof jaclaz Link to comment Share on other sites More sharing options...
deomsh Posted October 3, 2020 Author Share Posted October 3, 2020 7 hours ago, jaclaz said: Likely (but again it should be timed) something more like this (as well really "ugly") should be faster (less loops, hopefully only the needed ones): Interesting, but I will have to study the used Fn calls to understand more of it. Your latest approach seems to work, but only if the variable begins and end with a double-quote character! In my dd-approach this is not needed (and remaining double-quotes if used on the FATCOPY.G4B command-line will be not at begin or end of the variable containing the source path). 7 hours ago, jaclaz said: Yep, it needs to be timed, but it is likely that the write to (md) works faster, though (with all due respect ) you made it more complex (and slower) than needed with your dd loop (as it goes one char at the time). Are there more precise timing possibities with GRUB4DOS? My timer in FATCOPY.G4B has a minimum of whole seconds, which is far to much for comparisons needed here. Link to comment Share on other sites More sharing options...
jaclaz Posted October 4, 2020 Share Posted October 4, 2020 Well, only for the record, it wouldn't be that much a limitation to need to enclose the variable in quotes before calling the subroutine, but you are right that we can do "better". This is more what I had in mind and needs not any exoteric function (and it is - I believe - actually readable/understandable). !BAT setlocal set /A mdmem=0x3000 > nul set /A myskip=0x600000 > nul set myvar="something" with space and two spaces aND "" quotes" set myvarnq= set myoffets= echo -n > (md)%mdmem%+1 write (md)%mdmem%+1 %myvar%\0 cat --locate=\x22 (md)%mdmem%+1 | set myoffsets= cat --locate=\x22 --replace=\x00 (md)%mdmem%+1 cat (md)%mdmem%+1 | set myvarnq=%myvarnq% if not ""=="%myoffsets%" call :tokenize %myoffsets% set myvar goto :eof :tokenize set /A start=0x%1+1 > nul cat --skip=%start% (md)%mdmem%+1 | set myvarnq=%myvarnq% if not ""=="%2" shift && goto :tokenize goto :eof About timing, it is like in batch, you need to put the routine in a loop (x100, x1000, x10000, whatever) and then the seconds are enough. I would bet on this latter code, as there is no call to subroutine if there are no quotes and only as many calls as there are quotes in the variable. Try it and see if it fits the bill. jaclaz Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now