Possible to do this Batch without using a temporary file?

[Yzöwl]

Here's some batch script code, which you may find useful. It is based upon what I quickly read from the opening post, so I apologize if it is no longer valid.

What it is supposed to do, is to use the built-in CertUtil command, (Windows Vista+), to perform the MD5 hashes.

To use it, just run it with a single argument. If the argument is an existing directory, every file not named App.md5 in that directory and its subdirectories will be hashed. If it's an existing file argument and it isn't named App.md5, it will be hashed. If a non existing file or directory is passed, the script will treat it as if your argument was the current directory.

The output, to App.md5, will be the full file path preceded with a semicolon and the hash on the line below it.

@ECHO="%~a1"|"%__APPDIR__%FIND.EXE" "d">NUL&&(CALL :SUB "%~1" "/S" "%%%%#")||(
	IF NOT EXIST "%~1" (CALL :SUB "%CD%" "/S" "%%%%#"
		>NUL "%__APPDIR__%TIMEOUT.EXE" 3&GOTO :EOF
	)ELSE IF /I "%~nx1"=="App.md5" (
		ECHO Input file %~nx1 cannot be hashed, exiting...
		>NUL "%__APPDIR__%TIMEOUT.EXE" 3&GOTO :EOF)
	CALL :SUB "%~1" "" "%~1")
@"%__APPDIR__%TIMEOUT.EXE" 3 >NUL&GOTO :EOF
:SUB
@(FOR /F "EOL=|DELIMS=" %%# IN ('DIR /B%~2/A-D-L-S "%~1"^
	 ^|"%__APPDIR__%FINDSTR.EXE" /IV "\\App.md5$"')DO @FOR /F "DELIMS=" %%$ IN (
	'^""%__APPDIR__%CERTUTIL.EXE" -HASHFILE "%~3" MD5 2^>NUL^|FIND /V ":"^"'
)DO @SET "_=%%$"&ECHO ;%~3&CALL ECHO(%%_: =%%)>>"App.md5"

Note: If you wish to change your filename from App.md5 you'll need to change it in three places, (lines 4, 11 & 13).

Edited December 13, 2019 by Yzöwl
Added an extra couple of explanatory paragraphs.

[bphlpt]

Hey guys, sorry it's taken me so long to get back to this and respond.

First, @jaclaz
 

On 12/8/2019 at 2:56 PM, bphlpt said:

• Why the trailing space after !DlP! in - ("!DlP! ") - ?

On 12/9/2019 at 4:30 AM, jaclaz said:

3. That is the "clever" part , and you should think about it and answer yourself the question .

I guess I'm not that clever.    I even changed the line to...

FOR /F "tokens=2 delims=<>:/|?*" %%? IN ("!DlP! ") DO (ECHO ERROR- "%%?" -- "!Dlp!")
FOR /F "tokens=2 delims=<>:/|?*" %%? IN ("!DlP!") DO (ECHO ERROR- "%%?" -- "!Dlp!")
FOR /F "tokens=2 delims=<>:/|?*" %%? IN (" !DlP! ") DO (ECHO ERROR- "%%?" -- "!Dlp!")
FOR /F "tokens=2 delims=<>:/|?*" %%? IN (" !DlP!") DO (ECHO ERROR- "%%?" -- "!Dlp!")
FOR /F "tokens=2 delims=<>:/|?*" %%? IN ("!DlP!   ") DO (ECHO ERROR- "%%?" -- "!Dlp!")
FOR /F "tokens=2 delims=<>:/|?*" %%? IN ("   !DlP!   ") DO (ECHO ERROR- "%%?" -- "!Dlp!")
FOR /F "tokens=2 delims=<>:/|?*" %%? IN ("   !DlP!") DO (ECHO ERROR- "%%?" -- "!Dlp!")

...to see if I could see what the "clever" part was. For a "valid" input, it didn't matter, of course, since the FOR "failed", but even with an "invalid" input, such as - c:akjsL:dfh\kasl;dfj\jalsdkj - with an embedded colon, the output was effectively the same:

ERROR- "dfh\kasl;dfj\jalsdkj " -- "akjsL:dfh\kasl;dfj\jalsdkj"
ERROR- "dfh\kasl;dfj\jalsdkj" -- "akjsL:dfh\kasl;dfj\jalsdkj"
ERROR- "dfh\kasl;dfj\jalsdkj " -- "akjsL:dfh\kasl;dfj\jalsdkj"
ERROR- "dfh\kasl;dfj\jalsdkj" -- "akjsL:dfh\kasl;dfj\jalsdkj"
ERROR- "dfh\kasl;dfj\jalsdkj   " -- "akjsL:dfh\kasl;dfj\jalsdkj"
ERROR- "dfh\kasl;dfj\jalsdkj   " -- "akjsL:dfh\kasl;dfj\jalsdkj"
ERROR- "dfh\kasl;dfj\jalsdkj" -- "akjsL:dfh\kasl;dfj\jalsdkj"
_Dir="c:akjsL:dfh\kasl;dfj\jalsdkj\"

I understand the general operation of the line. Using "tokens=2" with the illegal characters listed as delimiters means that if there are no illegal characters in the input string, then there will be no second item to find, so the FOR will "fail" and the rest of the line will be ignored.

Yes, "%%?" picked up the different number of spaces after "!DlP!", but that didn't make any operational difference. So, why have the space? What is the "clever" part?

 

On 12/8/2019 at 2:56 PM, bphlpt said:

• It's OK to just throw away any extra quote marks in and around the path, add a missing trailing backslash, and assume the remainder is OK, but if any other illegal characters are found then it's a fatal error? I just try to be consistent.

On 12/9/2019 at 4:30 AM, jaclaz said:

5. As said the treating of double quotes is correct, ...

I don't think so. What I mean is that, for example, if you input the string - 
c":"j"ak"dj"\s"jl"l\"sl"aj"df"hl"k\"ls"aj"kd"\""""""""""""" - your code simply strips out all of the quotes, sees that what is left is valid, encloses it in a singe set of surrounding quotes - "c:jakdj\sjll\slajdfhlk\lsajkd\" - and calls it a day. But if you include one single other illegal character, the string is considered an error. To me, consistency would require either stripping out ALL illegal characters, which is not a trivial task I know, or to consider that any quote marks inside the string, or mismatched quotes outside it, are also considered to be an error. That's what I meant about being consistent. That's just my opinion, and I know that even if you try to deal with all of the complications that you can think of, like you mentioned in your response to (4), you can almost always come up with another improbable situation to trip up a code's execution if you try hard enough. But didn't you say:

...if there is an illegal character, then the path is not valid (it is illegal), so yes, if any of the no-no characters is found the only possibility is to have a fatal error.

 

Regarding the "fun" code
 

On 12/9/2019 at 4:30 AM, jaclaz said:

And now, for the fun of it, try to visualize the results of the following batch before running it:

@ECHO OFF
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
SET _Dir="C:\*"
CALL :is_expanded_path !_Dir!
SET _Dir="C:\/"
CALL :is_expanded_path !_Dir!
SET _Dir="C:\*/"
CALL :is_expanded_path !_Dir!

SET Head="C:\
SET Body=
SET Tail= /"
FOR /L %%? IN (1,1,12) DO (
SET Body=!Body!?
SET _Dir=%Head%!Body!%Tail%

CALL :is_expanded_path !_Dir! %%?
)
ECHO .

SET _Dir="C:\|/"
CALL :is_expanded_path !_Dir!

SET _Dir="C:\P*/"
CALL :is_expanded_path !_Dir!

SET _Dir="C:\*a?*/"
CALL :is_expanded_path !_Dir!

GOTO :EOF


:is_expanded_path
ECHO %2 "%~dp1" 

 

I understood some of this, but there definitely were a lot of unexpected things.

What I DID understand, sort of, I think: LOL 

I know that * is a wildcard for 0 or more characters, and ? is a wildcard for 1 character. For the DIR command, I thought that / usually designates an option and the next character says what the option is. And in the subroutine, "%~dp1" expands to the directory and path, ending with a backslash.

In the "loop", ? wildcards are added to the "body" one at a time. The output shows "C:\? \", and "C:\?? \", etc, even though "?" is an invalid character for a file or directory, until a directory is found. In my case that is "C:\AMD\", then that is the same output for the rest of the loop. So apparently it is the case of the first directory that is found, no matter how many "?" are used?

I don't understand why both "C:\*" and "C:\/" output "C:\", while "C:\*/" outputs "C:\$Recycle.Bin\" for me, which is the first directory, hidden, system, or not.

"C:\*a?*/" also outputs "C:\AMD\", which I can see since I think * can match zero characters. "C:\P*/" outputs "C:\pagefile.sys\" for me, which isn't a directory, but it is the first file or directory alphabetically starting with "p". "C:\|/" outputs "C:\|/", which again, is an invalid character.

And, another interesting thing is that if you remove the "/" or tail in all of the various tests, then every single one of them output "C:\".

So, please enlighten me, wise one.

By the way, "Powerhell" is definitely the correct spelling. My efforts with it have been abandoned, though your links were much appreciated.

 

 

On 12/9/2019 at 2:02 PM, Mcinwwl said:

What Os and what version of PS are you using? How exactly are you calling 7zip?

As earlier said, Povwershell 2.0 had some quirks when running external commands, and some were solved or worked around (workarounded?) in later versions. Too many possible root causes to guess.

 

Thanks for the interest and response, @Mcinwwl 

Windows 7 and PS 2.0. Yes, I discovered that I could update PS, but since I was possibly going to distribute my routine to others using systems with XP and later, I didn't want to require them to also have to update their system, and it wasn't code I wanted to have to provide. I could very easily make a batch routine to call 7-Zip, and call that batch from PS, and I did that, but it just seemed wrong. Maybe hypocritical, since I was perfectly willing to use other external software, but I didn't want to use PS to call batch to call that other software. I either wanted to just use batch, or just use PS. The appeal to me of PS was it's ability to easily modify a file or folder's date and do date comparisons. (I wanted to do this for synchronization purposes, not just changing the date to the current date.)

Anyway, I'm using 7z.exe with the following compression commands, given one after the other to the same directory in three passes:

7z.exe a "..\SomeFile.Ext" -ms=off -mx=5 -m0=lzma2 -xr@"7z.Exclude.lst" -xr@"7z.nc.lst" -xr@"7z.PPMd.lst"
7z.exe a "..\SomeFile.Ext" -ms=off -m0=PPMd -ir@"7z.PPMd.lst"
7z.exe a "..\SomeFile.Ext" -ms=off -mx=0 -ir@"7z.nc.lst"

... where "..\SomeFile.Ext" varies and is passed to the routine.

I tried everything I could think of, and tried all of the tips I read about, to no avail. Using PS 2.0 I just couldn't get it done.

I went back to batch and found a few options that worked to change a file's date. You can use "TOUCH.EXE" from the Windows 2000 Resource Kit, or use NirSoft's BulkFileChanger, which is what I ended up using. The capability to change the date of an individual file is only an option for recent versions of the program, and it's ability to do so via batch programs with variables (vs via its GUI) is not well documented. (You have to create a temporary configuration file with the date and time you want to be used, and the date and time must only be in the form "dd-mm-yyyy hh:mm:ss".) Once I figured all of that out, by trial and error, it worked very well.
 

 

On 12/13/2019 at 12:43 PM, Yzöwl said:

Here's some batch script code, which you may find useful. It is based upon what I quickly read from the opening post, so I apologize if it is no longer valid.

What it is supposed to do, is to use the built-in CertUtil command, (Windows Vista+), to perform the MD5 hashes.

Thanks for responding, @Yzöwl, I very much appreciate it. I always look forward to reading your batch code, even if it is sometimes written so efficiently it is difficult to do so.  I often learn something new from your code. I have never used CertUtil, so the demonstration of its use was good to see. I didn't end up utilizing it, for the following reasons:

1) When possible, I try to write batch code that can run on Windows systems of XP and newer, just for the principle, even though I personally use Windows 7.

2) The external program I am using to create my .md5 file, md5sum.exe, found by me a number of years ago, and separately by jaclaz, and discussed starting here - https://msfn.org/board/topic/162476-looking-for-a-vbs-script-to-generate-md5-of-a-file/page/2/?tab=comments#comment-1034865 - is small, works very well, and it is fast. It is also able to recurse the directory with a simple option switch. I don't remember why I thought I needed to recurse the directory manually. So my entire routine is now reduced to:

:f_RecurseMd5 %_DirNames%
PUSHD "%~1"&("PathTo\md5sum.exe" -r *.* >App.md5)&POPD
GOTO :EOF

It's so small, I've folded it into the rest of my routine and it is no longer a subroutine at all. Your routine, as written, is flexible, allowing hashing of an individual file, but you can see that with md5sum, you just need to remove the -r option and list the filename instead of *.* to have the same capability. Both routines output to the standard output, which can be redirected at will.

3) Since the .md5 file was meant to be included with the files that were hashed, I didn't think it was appropriate to have the entire path for each file, but rather only the relative path from inside the folder which is being hashed. The format of such files that I have most often seen is "like"

70a45e89a75d16fccb700a0e368ec541 *7z-x64.msi
bed4f5058624589d002afc8d306d1d9b *7z.msi

... etc. Having the hash first, and on the same line, makes the file easier to read, and this format is accepted by programs that I have used that can verify the hashes, such as TeraCopy, among others. I know that your code could be modified to meet this requirement, but by making it bigger, and imperceptibly slower, AFAIK.

4) I really liked the idea of using a capability built into the OS, if the other "issues" could be resolved, but very unlike what I expected, your code was SLOW. I changed your :SUB like so: (having turned ECHO OFF in the main routine, and calling it such that the console window remained open)

:SUB
FOR /F "EOL=|DELIMS=" %%# IN ('DIR /B%~2/A-D-L-S "%~1"^
     ^|"%__APPDIR__%FINDSTR.EXE" /IV "\\App.md5$"')DO FOR /F "DELIMS=" %%$ IN (
    '^""%__APPDIR__%CERTUTIL.EXE" -HASHFILE "%~3" MD5 2^>NUL^|FIND /V ":"^"'
)DO SET "_=%%$"&CALL ECHO(%%_: =%% *%%~#
PUSHD "%~1"&("PathTo\md5sum.exe" -r *.*)&POPD

... with your routine and the addition of md5sum both sending their output to the console to see "real time" how fast they are. It doesn't matter which order the two parts of the subroutine are in, the md5sum.exe part runs NOTICEABLY faster. By visual guess, at least two or three times faster.

As an aside, a couple of questions about your code.
1) While I realize that CERTUTIL.EXE, FIND.EXE, FINDSTR.EXE, etc are "external commands", why do you preface them with %__APPDIR__% (yes, I know that's where the external command programs are located) and include the .EXE? Any reason besides just good programming practice? I think this might be the first time I've seen FIND and FINDSTR referred to this way.
2) Any reason you seem to chose to turn off ECHO on the individual lines, instead of just using @ECHO OFF?
3) It might be related, but why the "(" in the @(FOR line? It seems extraneous.

 

I think that addresses all of the responses.

Cheers and Regards
 

[jaclaz]

Given the "delims=<>:/|?*", what happens if the last character is a delimiter?

I.e.: 

DIP=dfhkasldfjjalsdkj|

or 

DIP=dfhkasldfjjalsdkj?

 

About the "fun" code, some of the things that may happen  can - as you did - be explained, some other things of what happens in batch has no real *rational reason* or explanation, some are quirks, and you need to learn to live with them. 

Rob van der Woude has a few pages documenting "strange" or "unexpected" things that may happen in batch (and how you can in some cases take advantage of those):

https://www.robvanderwoude.com/altmsdosbatch.php

https://www.robvanderwoude.com/useless.php

https://www.robvanderwoude.com/clevertricks.php

jaclaz

Edited December 24, 2019 by jaclaz

[bphlpt]

6 hours ago, jaclaz said:

Given the "delims=<>:/|?*", what happens if the last character is a delimiter?

I.e.: 

DIP=dfhkasldfjjalsdkj|

or 

DIP=dfhkasldfjjalsdkj?

Aaahhh! Without the "extra" space, there wouldn't be a "2nd" item for the FOR to find and trip the ERROR part of the code. That is clever! Thanks for the explanation, and the links.

Cheers and Regards