On 10/4/2020 at 10:40 AM, jaclaz said:

Well, only for the record, it wouldn't be that much a limitation to need to enclose the variable in quotes before calling the subroutine, but you are right  that we can do "better".

I didn´t ment it being a problem, only that two more double-quote characters has to be removed. See first print-screen. BTW I changed variable to an example which can easily led to an odd number of quotes on the FATCOPY.G4b command-line (latest double-quote character AFTER last slash). First run is your original Function13-batch, in the second run myvar is set between extra double-quotes, third run is the batch with my dd-approach, but already including your write (md)%mdmem%+1 %myvar%\0

On 10/4/2020 at 10:40 AM, jaclaz said:

This is more what I had in mind and needs not any exoteric function (and it is - I believe - actually readable/understandable).

I'm impressed, cat --skip=%start% (md)%mdmem%+1 | set myvarnq=%myvarnq% is a great idea! It's working nice.

On 10/4/2020 at 10:40 AM, jaclaz said:

About timing, it is like in batch, you need to put the routine in a loop (x100, x1000, x10000, whatever) and then the seconds are enough.

I made a small timing-batch. Variable myvar is centralized (in the four called batch-files above temporarily commented out) and writing to screen in LOOPTEST.G4B only. I took out a part of my FATCOPY.G4B-timer, so don't use around midnight.

I gave the four different approaches to remove double-quotes some nice names. WDDQUOTE.G4B is my dd-approach, but with your use of write and also your use of cat --locate=\x22 (md)%mdmem%+1 > nul || goto :eof to locate a double-quote character, used to run /not run the sub-routine (instead of mine - which is more complicated). Your three batches are in historical order.

----------------------------------------------------------------------------------------------------
looptest.g4b
----------------------------------------------------------------------------------------------------
!BAT
set=*
debug off
pager off
set testg4b=%1
if not exist testg4b && echo && echo command-line: looptest.g4b FILE.G4B NUMBER && echo && goto :eof
set /a numtest=%2 > nul
if not exist numtest && echo && echo command-line: looptest.g4b FILE.G4B NUMBER && echo && goto :eof
if %numtest%==0 && echo && echo command-line: looptest.g4b FILE.G4B NUMBER && echo && goto :eof
set myvar="something" with space and two  spaces aND "" quotes"
#set "myvar=something with space and two  spaces aND quotes"
if exist myvar && echo myvar: %myvar%
set time0=%@time%
set /a loopnums=0 > nul

:testloop
if %loopnums%>=%numtest% goto :timing 
set /a loopnums=%loopnums% + 1 > nul
set myvarnq=
setlocal && call %testg4b%
endlocal && set myvarnq=%myvarnq%
goto :testloop

:timing
set time1=%@time%
set /a "timesec0=%time0:~0,2% * 3600 + %time0:~3,2% * 60 + %time0:~6,2%" > nul
set /a "timesec1=%time1:~0,2% * 3600 + %time1:~3,2% * 60 + %time1:~6,2%" > nul
set /a "copytime=%timesec1% - %timesec0%" > nul
echo myvarnq: %myvarnq%
echo Test time: %copytime% seconds <=> Number of tests run: %loopnums%
echo

----------------------------------------------------------------------------------------------------

---------------------------------------------------------------------
wddquote.g4b
---------------------------------------------------------------------
!BAT
#set myvar="something" with space and two  spaces aND "" quotes"
set /A mdmem=0x3000 > nul
echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0
cat --locate=\x22 (md)%mdmem%+1 > nul || goto :eof
call :noquotes
goto :eof

:noquotes
set seekbyt=
cat --locate=\x22 --number=1 (md)%mdmem%+1 | set seekbyt=
if not exist seekbyt && cat (md)%mdmem%+1 | set myvarnq= && goto :eof
set seekbyt=0x%seekbyt%
set /A seekbyt=%seekbyt%
set /A "skipbyt=%seekbyt% + 1"
dd if=(md)%mdmem%+1 of=(md)%mdmem%+1 bs=1 skip=%skipbyt% seek=%seekbyt%
goto :noquotes

-----------------------------------------------------------------------------------------------------

----------------------------------------------------------------
de_quote.g4b
----------------------------------------------------------------
!BAT
#set myvar="something" with space and two  spaces aND "" quotes"
set myvar=%myvar%
echo %@retval% | set myvarl=
set myvarnq=
set /a counter=0 > nul
call :de_quote
goto :eof

:de_quote
call set thischar=%^myvar:~%counter%,1%:
set /a counter=%counter%+1 > nul
if not \%thischar%==\": set myvarnq=%myvarnq%%%thischar:~0,1%
if ":"=="%thischar:~0,1%" set myvarnq=%myvarnq:~0,-1% 
if %counter%==%myvarl% goto :eof
goto :de_quote

----------------------------------------------------------------

----------------------------------------------------------------
function13.g4b
----------------------------------------------------------------
!BAT
set /A mdmem=0x3000 > nul
set /A myskip=0x600000 > nul
#set myvar="something" with space and two  spaces aND "" quotes"
echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0 > nul
cat --locate=\x22 --replace=: (md)%mdmem%+1
set myvar=
call :Function13
set myvarnq=%myvar%
goto :eof

:Function13
call Fn.13 %myskip% ":"
set /A myskip=%@retval% > nul
cat --skip=%myskip% (md)0+0x3001 | set mytoken=:
set /A myskip=%myskip%+%@retval% > nul
set myvar=%myvar%%%mytoken:~1%
call Fn.11 %myskip% ":" && goto :Function13
goto :eof

----------------------------------------------------------------

----------------------------------------------------------------
tokenize.g4b
----------------------------------------------------------------
!BAT
set /A mdmem=0x3000 > nul
set /A myskip=0x600000 > nul
#set myvar="something" with space and two  spaces aND "" quotes"
set myvarnq=
set myoffets=
echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0 > nul
cat --locate=\x22 (md)%mdmem%+1 | set myoffsets=
cat --locate=\x22 --replace=\x00 (md)%mdmem%+1
cat (md)%mdmem%+1 | set myvarnq=%myvarnq%
if not ""=="%myoffsets%"
call :tokenize %myoffsets%
goto :eof

:tokenize
set /A start=0x%1+1 > nul
cat --skip=%start% (md)%mdmem%+1 | set myvarnq=%myvarnq%
if not ""=="%2" shift && goto :tokenize
goto :eof

----------------------------------------------------------------

Tests are all run in VBox 6.0.18 r136238.

First print-screen below is testing while each batch-file has to be read from disk, so each run again.

I also tried loading the four batches with insmod (using insmod with LOOPTEST.G4B was only about one second faster). Load-order or loading all batches at once made no difference.

On 10/4/2020 at 10:40 AM, jaclaz said:

I would bet on this latter code, as there is no call to subroutine if there are no quotes and only as many calls as there are quotes in the variable.

Try it and see if it fits the bill.

And the Winner is: tokenize.g4b , your last creation.

But the dd-approach doesn't seem so bad in terms of speed, only 3 seconds slower, although I don't understand why this is a constant number if using /not using insmod.

BTW: I only used dd in FATCOPY.G4b because I had no better idea to solve my problems of cutting out /inserting characters in a variable. Never because of speed.

Edited October 5, 20205 yr by deomsh
Typo + removal of wrong pictures

What you should compare would be the "dd" approach against the "tokenize" one with a myvar with long text and only one double quote, i.e.

set myvar=something strupidly long and with a single double quote " placed around the middle of the string, this should choke the dd approach

The difference in timing should increase, the more double quotes are in the string the more loops the tokenize will take while the dd approach with only one quote should loop anyaway all n chars times.

I believe that the strings passed to the routine in real world will be (invented data):
1) 90% no quotes <- no difference between "dd" and "tokenize"
2) 9% 1 or two quotes <- advantage of "tokenize" over "dd"
3) 1% several quotes <- very slight advantage, if any at all, of "tokenize" over "dd"

and anyway 3 seconds over 50000 loops is a very small difference, but the real test would be on a real system (not a VM) where "real" disk latency, let's say on a slowish device such as an USB stick might make dd much slower.

jaclaz

Edited October 6, 20205 yr by jaclaz

10 hours ago, jaclaz said:

The difference in timing should increase, the more double quotes are in the string the more loops the tokenize will take while the dd approach with only one quote should loop anyaway all n chars times.

I do not understand why my dd-approach should loop more times than number of quotes in the variable. Each quote needs one :noquotes-loop (unless you mean inside the dd-command because I set bs=1).

10 hours ago, jaclaz said:

What you should compare would be the "dd" approach against the "tokenize" one with a myvar with long text and only one double quote, i.e.

set myvar=something strupidly long and with a single double quote " placed around the middle of the string, this should choke the dd approach

However, see the print-screens

Because 50.000 tests gave only a small difference (last digit is ±1 second), I raised the number of tests to 500.000

So the ´statement´ in variable myvarnq still holds approximately. With ONE double-quote character in myvar (instead of five), the calculated difference will be around ½ second per 50.000 runs.

Edited October 6, 20205 yr by deomsh
Correction

Yes, you are right, it is the opposite of what I said , the more quotes there are, the more should be the difference, I was thinking of the "cat --locate=\x22 --number=1 (md)%mdmem%+1 | set seekbyt=", you get just the first occurrence at each loop, while the "cat --locate=\x22 (md)%mdmem%+1 | set myoffsets=" gets all the quotes in one read, then you re-write the whole part up to the quote, my guess is that the cat --locate is very fast, whilst the dd might be a "slow" command, particularly with bs=1.

About insmod use, I don't know, I remember an issue with a grub4dos batch, some time ago, where insmod made a huge difference. 

I'll see if I can find a reference to it.

 

EDIT: Found it, never mind, it was about insmoding WENV:

http://reboot.pro/topic/17728-release-mbrviewg4b-a-bat-tool-for-grub4dos/

jaclaz

 

Edited October 7, 20205 yr by jaclaz

This could be a "midway" version:

!BAT
setlocal
set /A mdmem=0x3000 > nul
set /A myskip=0x600000 > nul

set myvar="something" with space and two  spaces aND "" quotes"
set myvarnq=
set myoffets=

echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0
cat --locate=\x22 (md)%mdmem%+1 | set myoffsets=
cat --locate=\x22 --replace=\x00 (md)%mdmem%+1
cat (md)%mdmem%+1 | set myvarnq=%myvarnq%
set ocount=0
if not ""=="%myoffsets%" call :noquotes2 %myoffsets%
set myvar
goto :eof


:noquotes2
set seekbyt=%1 > nul
if not exist seekbyt && cat (md)%mdmem%+1 | set myvarnq= && goto :eof
set /A seekbyt=0x%seekbyt%-%ocount% > nul
set /A "skipbyt=%seekbyt%+1" > nul
dd if=(md)%mdmem%+1 of=(md)%mdmem%+1 bs=1 skip=%skipbyt% seek=%seekbyt% > nul
shift
set /a ocount=%ocount%+1 > nul
goto :noquotes2

that should be good to compare the "pure" speed of cat vs. dd.

jaclaz

9 hours ago, jaclaz said:

Yes, you are right, it is the opposite of what I said , the more quotes there are, the more should be the difference, I was thinking of the "cat --locate=\x22 --number=1 (md)%mdmem%+1 | set seekbyt=", you get just the first occurrence at each loop, while the "cat --locate=\x22 (md)%mdmem%+1 | set myoffsets=" gets all the quotes in one read, then you re-write the whole part up to the quote, my guess is that the cat --locate is very fast, whilst the dd might be a "slow" command, particularly with bs=1.

Sounds logical, here are the tests. To raise the number of qoutes I invented a user who wants to be sure: every word, and every space between quotes. I changed myvar a bit, so the second double-qoute character is after the 16th position.

Indeed, difference has become huge. With a high number of qoutes my dd-approach is much slower then your tokenize-approach.

But sometimes weakness is strength: the number of characters to cut-out is virtual unlimited with the dd-approach.

Print-screen's this time in reverse order. Watch the last word in myvar and myvarnq:

The tokenize-approach seems to have a limit I first didn't understand, until I saw the memory-addresses (somehow I counted with two hex digits memory - but that's max 255 of cause) + one space - :

In a real filesystem this number of qoutes will of cause (almost) never exist, but there may be other uses.

9 hours ago, jaclaz said:

EDIT: Found it, never mind, it was about insmoding WENV:

http://reboot.pro/topic/17728-release-mbrviewg4b-a-bat-tool-for-grub4dos/

Yep, same reason why I am using insmod. In the mean time I counted the difference between two tests of quote-removing approaches, without or with insmod is approximately the same number of seconds (always ±1 second, because of the use of %@time% ! See print-screen´s in my post of last Monday 11:43).

Edited October 7, 20205 yr by deomsh
Typo

HELP while posting you already came with something new.

To be continued.

2 hours ago, jaclaz said:

This could be a "midway" version:

Nice, also the now needed correction of variable seekbyt with variable ocount.

2 hours ago, jaclaz said:

that should be good to compare the "pure" speed of cat vs. dd.

Hmm.

If I was a programmer (which I am not), I would say that MIDWAY.G4B call´s the cat-subroutine in GRUB4DOS (if fully optimized) the number of qoutes - 1 less, compared to WDDQUOTE.G4B. But let´s test (screen-writing and the set myvar-line commented out in all batches except LOOPTEST.G4B):

Because of the ±1 digit resolution, I raised the number of tests to 500.000 to get a measurable difference:

Your ´midway´ seems to be slower than needed: I thought you would have had this in mind?

!BAT
setlocal
set /A mdmem=0x3000 > nul
set myvar="something" with space and two  spaces aND "" quotes"
set myvarnq=
set myoffsets=
echo -n > (md)%mdmem%+1
write (md)%mdmem%+1 %myvar%\0 > nul
cat --locate=\x22 (md)%mdmem%+1 | set myoffsets=
#cat --locate=\x22 --replace=\x00 (md)%mdmem%+1
#cat (md)%mdmem%+1 | set myvarnq=%myvarnq%
set /a ocount=0 > nul
if not ""=="%myoffsets%" call :noquotes2 %myoffsets%
set myvar
goto :eof

:noquotes2
set seekbyt=%1 > nul
if not exist seekbyt && cat (md)%mdmem%+1 | set myvarnq= && goto :eof
set /A seekbyt=0x%seekbyt%-%ocount% > nul
set /A "skipbyt=%seekbyt%+1" > nul
dd if=(md)%mdmem%+1 of=(md)%mdmem%+1 bs=1 skip=%skipbyt% seek=%seekbyt% > nul
shift
set /a ocount=%ocount%+1 > nul
goto :noquotes2

Gives following results:

Seems to be faster than, or as fast as your earlier tokenize-approach (±1 digit).

What´s your ´conclusion´ about speed of cat vs dd?

I don't know, both?

The only difference between the midway and the midway "clean" seems to me (besides the set command for ocount, which I cannot believe can be relevant) the removal of the (unneeded) "cat --locate=\x22 --replace=\x00 (md)%mdmem%+1", so that it must take some time, but it is only run once, so?

In a "direct comparison", the "tokenize" *needs* it and with it the cat in the subroutine is faster than dd (QED ), BUT as you correctly did, by removing it  (as it is not needed in the dd version) the batch regains some speed, or maybe the difference is all in the insmod loading? 

Side note about the 255 chars limit, I don't see it as a real world limitation, though of course it is not "fully" universal.

I like to see this kind of stuff (when talking with "real" programmers, which would obviously consider it a serious flaw[1]) like doors .

The size of a door in an apartment is normally WxH 80 cm x 210 cm.

It has been de facto standardized as it is "wide enough" and "tall enough" to let 99,9 % (say) of people go through "comfortably", beside having a "pleasing to the eye" ratio.

If you are a basketball player or a seriously obese person they won't be good.

Still, we cannot make all doors - still say - WxH 120 cm x 250 cm to take care of the exception and if we add a control of some kind to prevent large people to go through (nowadays that would be an IoT device, a 3D scanner that assesses the size of the person approaching the door, sending it to a central server where an AI would  then - in case of need - return to a speaker either a "Duck!" or a "Stop right there, fatso!" [2] vocal warning) that would be inconvenient.

Anyway the exercise confirmed the base assumption, to know which is faster/better/whatever you need to test.

jaclaz

[1] typical is "you cannot use Excel for that, you need a proper database such as SQL", and BTW by the time they setup their "proper" database and the forms and the connection, I usually already have the results, printed and faxed to destination
[2] my imaginary IoT devices are not politically correct  
 

 

 

Edited October 8, 20205 yr by jaclaz

On 10/8/2020 at 10:26 AM, jaclaz said:

I don't know, both?

The only difference between the midway and the midway "clean" seems to me (besides the set command for ocount, which I cannot believe can be relevant) the removal of the (unneeded) "cat --locate=\x22 --replace=\x00 (md)%mdmem%+1", so that it must take some time, but it is only run once, so?

In a "direct comparison", the "tokenize" *needs* it and with it the cat in the subroutine is faster than dd (QED ), BUT as you correctly did, by removing it  (as it is not needed in the dd version) the batch regains some speed, or maybe the difference is all in the insmod loading? 

 

Sorry for the delay, but during testing smaller differences I got inconsistent results.

If one test-time is ±1 second, differences can be ±2 seconds, so ranges overlap easily (apart from load-time variations). I don´t like the ever longer time needed to do tests, so I made a new version of LOOPTEST.G4B. This took much more time than I expected, but nevertheless: I had a great time.

I managed to get a relatively stable time base and some semi-floating point operations to produce decimals. The stability depends upon the (virtual) hardware used.

Although I owe LimboX86 much for development-time on my Android smartphone, the time-base is very unstable, even in the ´whole second´ range. VBox 6 is better, but still unstable. Which is sad because of the productivity with ImDisk and the nice print-screens VBox is able to produce. I got my best results on a ASUS H61E board with Intel i3 (3,3 GHz) and DDR3 1333 memory (and I had to go back to GRAB.G4B. ).

BTW: all new test are done on the H61E-board, Grub4Dos is started from real-mode MS-DOS 7.1. So results can not be compared with the earlier tests in Vbox 6 (running under Windows 10 with AMD Bulldozer / DDR3 1666)!

This time I want to show my tests about the influence of insmod first, other comparisons will be done later. It´s also a very short introduction to the possibilities of the new version of LOOPTEST.G4B. The command-line of LOOPTEST.G4B has been changed a bit, to give room for arguments of a test G4B-file, besides other changes.

I compared MIDWAY.G4B and the cleaned version (three lines where not needed) with Short File Name MIDWAY#3.G4B. First I tested all files on a SATA flash drive, this is (hd0,0) on the first two print-screens. To compensate for internal delays in LOOPTEST.G4B I made a BIAS-variable. Further is variable 'myvar' set outside LOOPTEST.G4B (universal solution, names of input and output variable can be set on the command-line of LOOPTEST.G4B, and echoing can be done once, before and after the loops).

The results without and with insmod are different, the variations in load-time seems to be bigger than the runtime-differences.

On the next print-screen the same test on my slowest device, a very old 16MB M-SYS USB1.1 flash drive (my first one!).

Same results as with the SATA flash drive, but the insmod-tests seems to be very stable in the range of ±1 millisecond in both tests.

To minimize load-time instability, I did the same test on a Grub4Dos memdrive. The first two print-screens show the making of the memdrive. They are also a preview of my (forthcoming) utilities FATDEL.G4B and FATLSDIR.G4B. FATCOPY.G4B is showed with use of wildcard ´*´.

FAT cannot delete read-only files, so they will remain (until someone make a Grub4Dos ´attrib´-utility).

The last two print-screens show tests with number of digits set to three (by setting needed number decimals - rounding down). First test-cycle with 1000 tests does use ALMOST the same BIAS as earlier (the BIAS set during this test is slightly different; influenced by instabilities in the time-base and in the possibilities to divide the whole seconds.

Without and with the use of insmod all differences are in the range of 42-43 milliseconds.

I expanded the test to 10000, 100000 and 1000000 loops, always three digits only (last print-screen).

Without and with the use of insmod all differences are still in the range of 42-43 units, applicable to the different tests (0.01, 0.1 and 1 second respectively).

All in all, it seems to me that testing with insmod will give the ´right´, and the most stable differences. So next tests will be only with insmod, also to become independent of the device in use.

 

Side note.

When you output a "DIR" listing, you'd better "format" the strings to fixed size.

Usually this is done *like* (right align to length 12):

set myvar=something
#after the = in following line there are twelve spaces
set "myvar=            %myvar%"
set "myvar=%myvar:~-12%"

and (left align to length 12)
set myvar=something
#after the = in following line there are twelve spaces
set "myvar=%myvar%            "
set "myvar=%myvar:~0,12%"

jaclaz

On 11/8/2020 at 2:31 PM, jaclaz said:

Side note.

When you output a "DIR" listing, you'd better "format" the strings to fixed size.

Thanks!

FATLSDIR.G4B is not finished, but the appearance is important, especially for a DIR-program. I made some changes in between.

Better, or would you prefer the MS-DOS DIR-format? 

BTW: the FAT-based directory-parser gives date and time of files, so that's (relatively) easy. I managed to add filesize if the LS-directory parser is in use, ment for LFN and for other file-systems. Is it possible to get date and time of those files too?

Edited November 9, 20205 yr by deomsh
Typo

I don't think that the ls can have the dates/times (on non-fat), it is an internal command, it would need changes to grub4dos itself. 

What you got now in FATLSDIR.G4B seems to me very good "as is". 

Personally (but it is just an opinion/preference), I would:
1) shorten the batch name to FATDIR.G4B or DIRFAT.G4B or even FDIR.G4B
2) have the  (simplest) output similar to the ls output (only filenames, BUT with a filename per line) without switches
3) have the output with filename and  size (and directory total) with the /z switch (I know that it would make more sense to use the /s one, but that is usually associated with the normal DIR /s, where the s is short for recursive , and on the other hand in batch variable expansion %~z1 is size)
4) have the output with added date and time with the /v (verbose) or with the /dt (date-time) switch

jaclaz

14 hours ago, jaclaz said:

I don't think that the ls can have the dates/times (on non-fat), it is an internal command, it would need changes to grub4dos itself. 

 

Thanks for the information, so that will not be an option for me.

14 hours ago, jaclaz said:

Personally (but it is just an opinion/preference), I would:
1) shorten the batch name to FATDIR.G4B or DIRFAT.G4B or even FDIR.G4B

There is no internal reference to the filename, so the user can do what he or she wants, regarding renaming. The FATLSDIR.G4B HELP gives following suggestion: ´More convenient => insmod (bd)/fatutils/fatlsdir.g4b dir´ (say FATLSDIR.G4B is in that directory).

14 hours ago, jaclaz said:

2) have the  (simplest) output similar to the ls output (only filenames, BUT with a filename per line) without switches

So you prefer an output like DIR /B in MS-DOS (Dutch version)?

For now I added the /b-switch (both print-screens below). First print-screen shows the output of the FAT-based directory-parser, second one the LS-based on a NTFS-volume. Total size / files are removed too.

14 hours ago, jaclaz said:

3) have the output with filename and  size (and directory total) with the /z switch (I know that it would make more sense to use the /s one, but that is usually associated with the normal DIR /s, where the s is short for recursive , and on the other hand in batch variable expansion %~z1 is size)

I will think about it, at least there is the /b-switch now (´/B´ can be used too, all switches are case-insensitive).

BTW: The /s-switch is already in use, same output like DIR /s. I added the ´/s /q´-output to both print-screens as an illustration of the output of a whole drive without showing directories or files (sadly the first ´/´ is still needed to represent the root, without I ran into trouble if switches are in use).

Directory-output directly above the files can be removed too, but I will have to think a while about the consequences for rewriting my batch if /s is in use in conjunction with /b.

The /ls-switch is only explicitly needed to show full LFN´s on a FAT volume. On all other supported file-systems there is auto-switching to LS-mode (last print-screen above). I haven´t decided yet if its better to show this message in standard-mode, or hide it and add a switch.

Edited November 9, 20205 yr by deomsh
Typo's, correction

Yep , your way, "normal" DIR without switches and DIR /b to have "DIR /B" output  is also perfectly fine .

Maybe you could use (like in other grub4dos commands) the hyphen as separator for switches, i.e. -b, -s, -q, etc., otherwise something *loosely like* :

set param1=%1
if not "%param1:~-1,1%==/" <do something inserting a / as first parameter>

might do. this should "catch" lines where you omit the / (because in this case the second parameter (switch) is shifted to first) while keeping untouched the provided one, either a single / or a path (like (hd3,0)/) terminated with /.

jaclaz