How to pass/convert "http%3a%2f%2f" in batch variable

[bphlpt]

I wrote a script to help me download the latest DirectX redistributable from MS, attached below - Download.cmd - and now, of course, MS changed the coding on their page so that the link is now in the form of:

http%3a%2f%2fdownload.microsoft.com%2fdownload%2f8%2f4%2fA%2f84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14%2fdirectx_Jun2010_redist.exe

instead of what it was:

http://download.microsoft.com/download/8/4/A/84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14/directx_Jun2010_redist.exe

Anyone have any ideas if/how this coding can be handled in batch? Especially how to pass this - http%3a%2f%2fdownload.microsoft.com%2fdownload%2f - in a variable? I guess the best/most flexible way might be to try to find the link first using the friendly "http://download" type format, and if not found to change ":" to "%3a" and "/" to "%2f" and search again, and if found convert "%3a" and "%2f" back to ":" and "/" in order to end up with a usable link for wget. But I'm embarrassed to say that I've been away from batch coding long enough that I'm drawing a blank how best to handle all the manipulations of "%" required. Could anyone please remind me or help with the rewrite?

Here is the ReadMe for the file:

Download -

Written By - bphlpt - Well, mostly by dumpydooby, with influences by vmanda

and Pasha_ZZZ, while cheered on by 5eraph, alban, and

user_hidden, while YumeYao waved from the background.

Version - 1.5.0

Date - 2010-11-01

Download "%_DownloadPage%" "%_LinkStart%" "%_LinkEnd%" "%_DownloadWhere%"

["%_StatusOut%"]

The first four argumnets are Required, and the fifth is optional. They are:

"%_DownloadPage%" - The web page where the download link is located.

"%_LinkStart%" - The first part of the link you want to match.

"%_LinkEnd%" - The last part of the link you want to match.

"%_DownloadWhere%" - Where the downloaded file should be stored.

"%_StatusOut%" - This can be any valid redirection path - filename, the

console, or even nul. If this argument is omitted, the

StdOut, or console, is used.

NOTE: Each argument should be enclosed with quotes ["]

This program finds a download link specified as "%_LinkStart%***%_LinkEnd%"

on the web page "%_DownloadPage%" and downloads that file to

"%_DownloadWhere%" using wget. It has been tested to work using XP and Win7

specifically for Microsoft's DirectX download page.

This program began from my need for this funtion by my "DirectX file

Collector and addon Builder", see here -

"http://www.ryanvm.net/forum/viewtopic.php?p=108233". dumpydooby suggested

adding the function, and after I was originally unsuccessful in its

implementation, vmanda came up with a great tool that filled the need. But I

was stubborn and still wanted to do my own version. As I was just starting

to experiment with VBScript, Pasha_ZZZ provided a perfect tool using JScript.

But before I had time to roll out the DXCB including Pasha_ZZZ's great code,

dumpydooby came back with essentially the same code in VBScript but with

enhanced wget funcionality. That became the basis for this code. I merely

shined it up a bit, generalized it, added a variable output stream, added

error checking and documentation, and corrected a few "gotcha's" we've all

learned from. 5eraph, alban, user_hidden, and YumeYao helped immeasurably

with testing. This would not have been possible without the help of

everyone mentioned.

Dependencies: wget must be located in the same directory as this file and the

first four arguments specified above must be defined.

Upon Exit, ERRORLEVEL is set to 0 if all went well, 1 if no links were found,

and 2 if there was a problem trying to download the link with wget.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is an example of a way to use Download:

@ECHO OFF & SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION

(SET _DownloadPage="http://go.microsoft.com/fwlink/?LinkID=56511&clcid=0x409")

(SET _LinkStart="http://download.microsoft.com/download/")

(SET _LinkEnd="_redist.exe")

(SET _DownloadWhere="%CD%")

(Call Download %_DownloadPage% %_LinkStart% %_LinkEnd% %_DownloadWhere% con)

Exit /B 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Any download link in the format "%_LinkStart%***%_LinkEnd%" that is found on

"%_DownloadPage%" will be downloaded by wget and put "%_DownloadWhere%" unless

the file already exists there and has a date code no older than that of the file

on the "%_DownloadPage%". In this example, status will be provided on the

StdOut, or "con", including a running status in the Title bar.

The download includes:

Download.cmd

wget.exe

ReadMe.txt

Changelog:

2010-11-01 -- Version 1.5.0 -

Improved reliability of Downloader with code tweaks

2010-09-13 -- Version 1.0.0 -

Initial Release

Thanks in advance for any and all suggestions anyone can provide.

Cheers and Regards

Download.cmd

[jaclaz]

That's not "different".

For *any* reason *something* changed the link.

Those are "common" characters "translated".

%3a=:= ASCII 58 Dec

%2f=/= ASCII 47 Dec

etc.

Managing % signs in batch is tricky, but in this case it can be done nonetheless, by using a subroutine.

You can use Variable text substitution allright, example:

CALL :my_sub §1 §2 §3 §4 §5 §6 §7 §8 §9



GOTO :EOF

:my_sub
SET myaddress="http%3a%2f%2fdownload.microsoft.com%2fdownload%2f8%2f4%2fA%2f84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14%2fdirectx_Jun2010_redist.exe"
SET myaddress=%myaddress:§3a=:%
SET myaddress=%myaddress:§2f=/%
SET myaddress
GOTO :EOF

See if something like the above fits in your script/for your use.

jaclaz

EDIT: beware of trailing spaces added by the board!

Edited June 10, 2011 by jaclaz

[bphlpt]

Thanks for the reply jaclaz. i always learn something from reading your posts.

I knew that the two versions of the line I posted were "equivalent" though not equal, being familiar with ASCII.

Trying to learn more, in the line:

CALL :my_sub §1 §2 §3 §4 §5 §6 §7 §8 §9

what are all the "§1 §2" etc for? They look like parameters, but what exactly is being passed? And how do you make the "§" character and what does it mean?

Then in the code:

SET myaddress=%myaddress:§3a=:%

SET myaddress=%myaddress:§2f=/%

I sort of know how variable substitution works, but I didn't expect to see the "§" character - again, what does it mean?

Running the code I can see that it works, but I want to know WHY it works to better understand how to use/modify it.

Re modifying it...

The download.cmd routine is called passing "%_DownloadPage%", "%_LinkStart%" and "%_LinkEnd%" to it. It, in turn uses wget to get the Server Response from the "%_DownloadPage%", and then get the "%_ConfirmationFile%" that is specified there. Then it creates a VB script routine embedding the "%_LinkStart%" and "%_LinkEnd%" in it and calls that VB script routine to parse the "%_ConfirmationFile%" looking for "%_LinkStart%***%_LinkEnd%" and returning all the unique links it finds on that page which match that pattern.

Here's the Call to create the VB script, run the routine, and get the first link found, if any:

CALL :sub_MakeVBS

>%_DownloadLoc% cscript //NoLogo %_ParseVbs% %_ConfirmationFile%

(SET /P _FirstLoc=<%_DownloadLoc%)

IF NOT DEFINED _FirstLoc GOTO s_NotFound

And here's where it actually creates the VB script:

:sub_MakeVBS

>%_ParseVbs% (

ECHO.Option Explicit

ECHO.Dim fso, fs, d, regEx, Matches, Match, strKey

ECHO.Set fso = CreateObject("Scripting.FileSystemObject"^)

ECHO.Set fs = fso.OpenTextFile(WScript.Arguments(0^),1,False^)

ECHO.Const TextMode = 1

ECHO.Set d = CreateObject("Scripting.Dictionary"^)

ECHO.d.CompareMode = TextMode

ECHO.Set regEx = New RegExp

ECHO.With regEx

ECHO. .Pattern = "%_LinkStart:.=\.%(\S*?)%_LinkEnd:.=\.%"

ECHO. .Ignorecase = True

ECHO. .Global = True

ECHO.End With

ECHO.Set Matches = regEx.Execute(fs.ReadAll^)

ECHO.fs.Close

ECHO.For Each Match in Matches

ECHO. If Not d.Exists(Match.Value^) Then

ECHO. d.Add Match.Value , 0

ECHO. End If

ECHO.Next

ECHO.For Each strKey In d.keys

ECHO. WScript.Echo strKey

ECHO.Next

)

EXIT /B 0

So, with all the steps currently involved, it SEEMS that it would make sense to use the "friendly" version of the link parts as much as possible and only do the change as "deep" in the code as I can to minimize the hassles in trying to pass the "%" character. That would mean doing it in the created VB script, ie in "regEx.Pattern". But I've forgotten whether I can modify .Pattern directly or if I need to create a new .Pattern from modified "%_LinkStart%" and "%_LinkEnd%" elements? And do I need to Close fs and reopen it in order to reread it? I've been away from this stuff for too long. LOL

I would like to end up with the following logic, all in the VB script that I'm creating and within the batch download.cmd (At least that seems to make the most sense of where to put it):

...

ECHO.Set regEx = New RegExp

ECHO.With regEx

ECHO. .Pattern = "%_LinkStart:.=\.%(\S*?)%_LinkEnd:.=\.%"

ECHO. .Ignorecase = True

ECHO. .Global = True

ECHO.End With

ECHO.Set Matches = regEx.Execute(fs.ReadAll^)

ECHO.fs.Close

...

***If there aren't any matches, then recreate regEX with all ":" and "/" changed to "%3a" and "%2f", then reopen fs and re-search:

...

ECHO.Set Matches = regEx.Execute(fs.ReadAll^)

ECHO.fs.Close

...

***If any matches are found now, change all "%3a" and "%2f" back to ":" and "/" in those matches. Then:

...

ECHO.For Each Match in Matches

ECHO. If Not d.Exists(Match.Value^) Then

ECHO. d.Add Match.Value , 0

ECHO. End If

ECHO.Next

...

This way, all manipulation of the link parts can be in "friendly" form, and I won't have to change the code again to be able to handle links in either form, since it will look for both types if necessary. And all of this will be transparent to the rest of download.com and the calling program. Can anyone help me with this code, or suggest an alternate approach to accomplish this goal?

Thanks in advance again to all help anyone can provide.

Cheers and Regards

[allen2]

Someone already wrote one function to do url decode in vbs.

[jaclaz]

The "§" is just a "never or very rarely used" character I have on my italian keyboard, it's ALT+0167

You can use any other character that is "never" used that you have handy, like "#" or "Ø" or a string that you know is not in the address.

As an example you could do (since in the example we have only %2f and %3a:

CALL :my_sub a_suffusion_of_yellow minnie mickey_mouse

and later:

SET myaddress=%myaddress:mickey_mousea=:%

SET myaddress=%myaddress:minnief=/%

Whenever the command parser finds a "%" followed by a number, like "%2", "%3" since that is the way to designate parameters in batch, it automatically sustitutes them with the value of the corresponding parameter.

%0 is the actual command that started the batch

%1 to %9 are the first nine parameters passed to it when invoked.

So if you do not pass these parameters when the batch finds:

"http%3a%2f%2fdownload.microsoft.com%2fdownload%2f8%2f4%2fA%2f84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14%2fdirectx_Jun2010_redist.exe" 

it will nicely "expand" it to:

"httpaffdownload.microsoft.comfdownloadf8f4fAf84A35BF1-DAFE-4AE8-82AF-AD2AE20B6B14fdirectx_Jun2010_redist.exe" 

i.e. will replace any occurrence of %2 and of %3 with the values of the corresponding parameter (which is "null").

by calling a sub-routine giving it the (fake) parameters, we replace effectively %2 and %3 with the values of the respective parameter, i.e. §2 and §3.

Then we replace §2f with / and §3a with : .

jaclaz

Edited June 11, 2011 by jaclaz

[bphlpt]

Thanks, jaclaz! the explanation is very clear - very helpful.

@allen2 - I appreciate the link, but I'm afraid I don't understand the routine. The only parameter, "What", I assume is the web page you want to parse essentially passed as a single string? And if there are more than one link on the page, how do you specify which one you want? Or did I totally misread how this works?

Cheers and Regards

[Yzöwl]

Just to expand on jaclaz's idea a little, here's a more lengthy script which includes some of the more common 'translations'

@ECHO OFF
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
SET "_url=%*"
CALL SET "_url=!_url:%%=§!"
SET "_url=!_url:§2A=*!"
SET "_url=!_url:§27='!"
SET "_url=!_url:§28=(!"
SET "_url=!_url:§29=)!"
SET "_url=!_url:§3B=;!"
SET "_url=!_url:§3A=:!"
SET "_url=!_url:§40=@!"
SET "_url=!_url:§26=&!"
SET "_url=!_url:§3D==!"
SET "_url=!_url:§2B=+!"
SET "_url=!_url:§24=$!"
SET "_url=!_url:§2C=,!"
SET "_url=!_url:§2F=/!"
SET "_url=!_url:§3F=?!"
SET "_url=!_url:§23=#!"
SET "_url=!_url:§5B=[!"
SET "_url=!_url:§5D=]!"
SET "_url=!_url:§3C=<!"
SET "_url=!_url:§3E=>!"
SET "_url=!_url:§7E=~!"
SET "_url=!_url:§2E=.!"
SET "_url=!_url:§22="!"
SET "_url=!_url:§7B={!"
SET "_url=!_url:§7D=}!"
SET "_url=!_url:§7C=|!"
SET "_url=!_url:§5C=\!"
SET "_url=!_url:§2D=-!"
SET "_url=!_url:§60=`!"
SET "_url=!_url:§5F=_!"
SET "_url=!_url:§5E=^!"
SET "_url=!_url:§25=%%!"
SET "_url=!_url:§20= !"
SETLOCAL DISABLEDELAYEDEXPANSION
CALL SET "_url=%%_url:§21=!%%"
SET _url
PAUSE

[jaclaz]

@Yzöwl

The masterpiece is obviously the:

SETLOCAL DISABLEDELAYEDEXPANSION 
CALL SET "_url=%%_url:§21=!%%" 

jaclaz