How to make list of files(tree) include files in *.zip files?

[ruthan]

Hello,
i would like to learn how to make list of files from some directory, include subfolders(infinite) level with content of .zip files include, add what is not .arj and *.rar and even. *.tar and other archives wold be nice too, but is not necessary, zip are primary.. I also dont want second level archives in archive.. and it would be nice to exclude list of *.zip from list.. I only need to see in which archive is which file.

 Its possible, does someone already have that code?

I can do it by brute force, extract files, move zip aside and call tree.. but its annoying, i no that FreeCommander and other can search inside archives, but i dont know how to report result of search (*.*)..

[hotnuma]

You can list files this way in python :

https://stackoverflow.com/questions/2212643/python-recursive-folder-read

A simplest example :

import sys, os

fromdir = "D:\\DevBuild"

for root, subs, files in os.walk(fromdir):
    for fname in files:
        fpath = os.path.join(root, fname)
        print(fpath)
    

That simple program prints all file paths in the specified directory.

To list the content of archives, a good way maybe to use the command line version of 7zip or bsdtar.

A more advanced example :

import sys, os, subprocess

def archlist(fpath):
    cmd = "C:\\Programs\\Outils\\7-Zip\\7z.exe", "l", "-ba", fpath
    
    subproc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    out = subproc.communicate()[0].decode('windows-1252').strip().split('\r\n')
    
    for line in out:
        print(line)


def dirlist(fromdir):
    for root, subs, files in os.walk(fromdir):
        for fname in files:
            fpath = os.path.join(root, fname)
            if fpath.endswith(".zip"):
                archlist(fpath)
            
    
dirlist("D:\\DevBuild\\parse")

This one runs 7z.exe from 7Zip, it redirects the output to a text buffer, and prints that buffet, so in my example, I get something like this :

2018-07-30 09:28:58 D....            0            0  b
2018-07-30 09:28:53 ....A            0            0  b\a.txt
2018-07-30 08:38:22 ....A          266          121  b\a.zip
2018-07-30 09:29:08 D....            0            0  b\c
2018-07-30 09:29:05 ....A            0            0  b\c\c.txt
2018-07-30 09:29:18 D....            0            0  b\d
2018-07-30 09:29:14 ....A            0            0  b\d\d.txt

The output of 7zip may not be really nice, for example it shows empty directories, a program like bsdtar gives cleaner output I think.

It's possible to do the same in Qt, it would run probably faster. I really like Qt, it's fabulous, but it's harder to learn.

So, that's a starting point.  

Edited July 30, 2018 by hotnuma

[ruthan]

Thanks, could you point be some good portable version of Python and how to run such script with it?

 Here is see 4 versions:
http://portablepython.com/

, i dunno if it will not have some problems with syntax details in post above..

  QT - I dont need understand how it works, i just need something to run and post valid results:)

Edited July 30, 2018 by ruthan

[hotnuma]

I use Python 3.4 under XP, but it should work with latest release from python.org :
https://www.python.org/downloads/

".py" files will be registered on the system and double clicking a ".py" files should run it.
The samples I posted should work but you'll have to change hard coded paths of course. 

[jaclaz]

Simple batch, using info-zip unzip (version 6.00):

@ECHO OFF
::ZipDir.cmd by jaclaz v0.001 31/07/2018
SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
IF %1.==. (SET DirPath="%~dp0") ELSE (SET DirPath="%~1\")
SET DirPath=%DirPath:\\=\%
IF NOT EXIST %DirPath% SET DirPath="%~dp0"
ECHO Directory of %DirPath%
ECHO   Size     Filename
ECHO --------- ------------------------------------------
FOR /F "tokens=* delims=" %%A in ('dir /b /s %DirPath%') do (
SET Size=         %%~zA
SET Size=!Size:~-9,9!
ECHO !Size! %%~dpnxA
IF %%~xA==.zip ECHO 	  ³&CALL :listzip "%%~dpnxA"
)

GOTO :EOF

:listzip
FOR /F "skip=2 tokens=* delims=" %%B IN ('unzip -l -q %1') DO (
SET Thisline=%%B
IF "!ThisLine:~0,4!"=="----" ECHO.&GOTO :EOF
REM IF "!ThisLine!"==" --------                   -------" ECHO.&GOTO :EOF
ECHO !ThisLine:~0,9! Ã!ThisLine:~28,50! 
)

jaclaz

 

zipdir.zip

[ruthan]

Hotnuma: I have problem to define searched string - filename, i have tried, to define fname="large.jpg", but its not right way.
  If its possible, could be dirlist - defined in "windows form" without double "'\"?
 How to use wild cards in filename? Or *filename*.* will work by default?
 I also some wait to key at the end would be nice - i would import msvcrt as m  in header + def wait():  m.getch() .. but it not seems to be working, im doign something wrong..

jaclaz: Thanks, but i would need some explanation, comments etc too.. I dont uderstand where is defined searched filename.. wildcards would be again too.. and i see some target directory definition,

but again standart without patch as input variable without any additional syntax would be nice too.

 I dunno if it matter but i would like to this tool for network paches too..  - For path as "\\10.0.0.15\!nas\Porn\" etc..

 

 

Edited July 31, 2018 by ruthan

[jaclaz]

10 hours ago, ruthan said:

jaclaz: Thanks, but i would need some explanation, comments etc too.. I dont uderstand where is defined searched filename.. wildcards would be again too.. and i see some target directory definition,

but again standart without patch as input variable without any additional syntax would be nice too.

 I dunno if it matter but i would like to this tool for network paches too..  - For path as "\\10.0.0.15\!nas\Porn\" etc..

 

 

There is no searched filename, as it is not what you asked:

On 7/30/2018 at 1:09 AM, ruthan said:

Hello,
i would like to learn how to make list of files from some directory, include subfolders(infinite) level with content of .zip files

what I posted is essentially a DIR equivalent comprising contents of ZIP files.

Now you are asking something different, a "search" feature for files either "plain" or within .zip archives.

If you want to search the output, and have a "full" path returned you will need to change the output prepending the path to the archive to all files within the zip  and pipe it into FIND or FINDSTR.

Should work for network paths as well, however.

Make a directory *like* C:\zipdir and copy to it the script and the unzip.exe, navigate to it, open a command prompt and type in it:

zipdir <path>

of course if <path> contains spaces you should enclose it in double quotes.

In the case of a network path, you need to add a backslash at the beginning (temporarily, the script can be easily fixed to take care of the leading \\ in network paths), see attached image.

jaclaz

[jaclaz]

Fixed:

@ECHO OFF
::ZipDir.cmd by jaclaz v0.002 01/08/2018
SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
IF %1.==. (SET DirPath="%~dp0") ELSE (SET DirPath="%~1")
IF NOT %DirPath:~-2,1%==\ SET DirPath="%DirPath:"=%\"

IF NOT EXIST %DirPath% SET DirPath="%~dp0"
ECHO Directory of %DirPath%
ECHO   Size     Filename
ECHO --------- ------------------------------------------
FOR /F "tokens=* delims=" %%A in ('dir /b /s %DirPath%') do (
SET Size=         %%~zA
SET Size=!Size:~-9,9!
ECHO !Size! %%~dpnxA
IF %%~xA==.zip ECHO 	  ³&CALL :listzip "%%~dpnxA"
)

GOTO :EOF

:listzip
FOR /F "skip=2 tokens=* delims=" %%B IN ('unzip -l -q %1') DO (
SET Thisline=%%B
IF "!ThisLine:~0,4!"=="----" ECHO.&GOTO :EOF
ECHO !ThisLine:~0,9! Ã!ThisLine:~28,50! 
)

jaclaz

Zipdir002.zip

[jaclaz]

And now, for no apparent reason , zipfind.cmd:

@ECHO OFF
::ZipFind.cmd by jaclaz v0.001 01/08/2018
SETLOCAL ENABLEEXTENSIONS
SETLOCAL ENABLEDELAYEDEXPANSION
IF %1.==. (SET DirPath="%~dp0") ELSE (SET DirPath="%~1")
IF NOT %DirPath:~-2,1%==\ SET DirPath="%DirPath:"=%\"

IF NOT EXIST %DirPath% SET DirPath="%~dp0"
SET LookFor=%2
IF NOT DEFINED LookFor SET LookFor=.

ECHO Looking for pattern "%LookFor%" in files in %DirPath%&ECHO.
FOR /F "tokens=* delims=" %%A in ('dir /b /s %DirPath%') do (
REM ECHO %%~dpnxA
IF %%~xA==.zip CALL :listzip "%%~dpnxA"
)

GOTO :EOF

:listzip
FOR /F "skip=2 tokens=* delims=" %%B IN ('unzip -l -q %1') DO (
SET Thisline=%%B
IF "!ThisLine:~0,4!"=="----" ECHO.&GOTO :EOF
ECHO %~1 !ThisLine:~28,50! | FINDSTR /I %LookFor%
)

 

zipfind001.zip

Edited August 1, 2018 by jaclaz

[ruthan]

20 hours ago, jaclaz said:

There is no searched filename, as it is not what you asked:

 Its in last sentence, even last work is search.., but whatever, i forgot to emphasize it. Whole sense of this is print which files in which archives match search patttern.. I not need to search within files, i need only list of filenames.

  So i need 2 input variables:
1) directory were i want to search (and all subdirectories levels), it is not defined, it would be nice to use local - were command was executed..
2) Searched filename - with classic dos wildcards support "?*"
   Output should be similar what you posted above, but i need only list files which match pattern and their full patch (path to zip and patch whin the zip).. File details are nice (especially size and date) but not necessary.

  Yeah if it would not have variable 2 option (i will not change code by myself), i can use your output as input of other tool like sed which will list only lines with searched string.. But it would means 2 files instead of 1 for execution and third file would be some temp files at least. and i dont could that additional searching tool.

  Im using for example for searching which programs in huge Dos / Amiga collection (packed to save space) are using same libraries or drivers, which i know that are problematic on some machines (typically too fast), or within some emulators.. so i find such library i want to know other programs,  which are using it.. more hits, more reasons to try to fix.

  

Edited August 2, 2018 by ruthan

[jaclaz]

The zipfind. cmd posted just above your last reply EXACTLY:

1) allows as parameters the root directory to search into and (optionally) a pattern to look for
2) prints the full path to the zip file containing the matched pattern file and the filename matching the pattern
3) works also for network paths

Since it uses FINDSTR, the use of wildcards is different from the "normal" ones, and matches the (sort of[1]) regular expression used in FINDSTR, of course you can replace in the script the FINDSTR with whatever similar tool that is pipable, such as grep.

jaclaz

 

[1] See:

https://blogs.msdn.microsoft.com/oldnewthing/20151209-00/?p=92361

[ruthan]

I have found out how to it with FreeCommander:

Problem was that top menu is active and specific to search and there such options.. which are not in icon bar, design is cumbersome, but feature are there.

- Select all search result and use Alt+Ins for "Edit - Copy path+name as text"
- Open any editor paste the clipboard content

Jaclaz : Thanks, but i spend quite time to try to read your script, but code without comments sucks, maybe its good for some Powershell / Admin guy, but not for me.. and i do have general programming knowledge.
  I really need to point out where is my input variable and which format..
    I ok i find out now probably - there probably %1, %2 so you have to set it by cmdline not inside the script.. Wildcard, i really need 2 of them * and ? i should be hard to translate from to FINDSTR ones..

[jaclaz]

1 hour ago, ruthan said:

Jaclaz : Thanks, but i spend quite time to try to read your script, but code without comments sucks, maybe its good for some Powershell / Admin guy, but not for me.. and i do have general programming knowledge.
  I really need to point out where is my input variable and which format..
    I ok i find out now probably - there probably %1, %2 so you have to set it by cmdline not inside the script.. Wildcard, i really need 2 of them * and ? i should be hard to translate from to FINDSTR ones..

Well, you posted:

On 7/30/2018 at 1:09 AM, ruthan said:

Hello,
i would like to learn how to make ...

The little script above is a plain, simple, batch script, nothing particularly fancy, and definitely NOT Powershell, I don't think I EVER wrote anything so clear/self explaining.

Essentially the variables are:

%DirPath% <- comment: this is the directory path where you wish to search it is passed to the batch as first argument or %1

%LookFor% <- comment: this is the pattern to look for it is passed (optionally) to the batch as second argument or %2

The line in the batch:

ECHO Looking for pattern "%LookFor%" in files in %DirPath%

should have been self-explanatory.

 

Abstracting from the minutiae, the batch is made of 3 (three) blocks of code:

1) do some minimal input validation:

IF %1.==. (SET DirPath="%~dp0") ELSE (SET DirPath="%~1")
IF NOT %DirPath:~-2,1%==\ SET DirPath="%DirPath:"=%\"

IF NOT EXIST %DirPath% SET DirPath="%~dp0"
SET LookFor=%2
IF NOT DEFINED LookFor SET LookFor=.

2) look for any file in with extension .zip in the path provided 

ECHO Looking for pattern "%LookFor%" in files in %DirPath%&ECHO.
FOR /F "tokens=* delims=" %%A in ('dir /b /s %DirPath%') do (
REM ECHO %%~dpnxA
IF %%~xA==.zip CALL :listzip "%%~dpnxA"
)

3) if such a file is found, use unzip to list its contents and pass these contents to the FINDSTR command:

:listzip
FOR /F "skip=2 tokens=* delims=" %%B IN ('unzip -l -q %1') DO (
SET Thisline=%%B
IF "!ThisLine:~0,4!"=="----" ECHO.&GOTO :EOF
ECHO %~1 !ThisLine:~28,50! | FINDSTR /I %LookFor%
)

All the "magic" happens in this line where the filename found inside the zip is piped to FINDSTR:

ECHO %~1 !ThisLine:~28,50! | FINDSTR /I %LookFor%

FINDSTR is a documented command, its syntax is similar to REGEX, but not exactly it, a said:

https://ss64.com/nt/findstr.html

and can be replaced in the last line of the batch by any similar pipable command. like as said grep.

If you have specific questions, by all means ask them.

jaclaz