Python 3.5 Runtime Redistributable backported to XP

[Dibya]

17 hours ago, dencorso said:

Some of our members (with all due respect) indulge in the latest version fetish (LVF), which can be mostly harmless when not taken to extremes. I, myself, do indulge in it sometimes, I reckon, but the farthes I usually go is to change the minimum "Subsystem Version" to allow executables to run, nowadays. If I really have something tha only runs on Win 7+, I either do it on 7SP1 or eschew it, usually the latter. As of today, all that I need to do in my day-to-day can be done in XP SP3. I consider the Extended Kernel approach mostly useless, although I dedicate lots of respect to @Xeno86, @jumper, and @blackwingcat.    I consider the WDMSYS/WDMEX (Extended Driver Interface) very useful, but nobody has been ever able to port to the NT-family OSes what Walter Oney and @rloew ( ) did for 9x/ME... and that's why we still haven't working Intel USB 3.0 drivers at this point in time. Of course, these are just my 2¢ and everybody may disagree. Moreover, YMMV, obviously. But, considering @Thomas S. did ask the queestion above, I felt it shouldn't remain unanswered. That's all.

Sage words and attitude, IMO. +1

It depends, IMO: people I know (even if just through the internet) and trust I don't consider foreign sources.
Otherwise I do agree to that, too.

You forgot wildBill , the man who invented petools and made kernel patches for NT os first time .

Without his tool, no one can extend section of a nt pe kernel without bsod (addition of section is different things)

His tool can add section even before RSC and relocation table.

I am angry

[Mathwiz]

The state of play, I believe, is: We have ProxHTTPSProxyMII 1.5 working with Cryptography 2.4.2, which runs even under Python 3.4. For now, this works well.

But TLS continues to evolve, so to avoid eventual obsolescence we'd like to get ProxHTTPSProxyMII working with newer Cryptography versions.

This required a newer version of Python, so we've got Python 3.7.1 working on XP; but more is required as well (at least a user-written version of bcrypt.dll to substitute for the Win 7 module, but apparently even more than that).

The problem is, once you move beyond Python 3.4, developers assume you're running on a newer Windows version, so all packages developed for newer Python versions may also use newer Windows functions. It may not be practical to go much further.

[dencorso]

3 hours ago, Dibya said:

You forgot wildBill , the man who invented petools and made kernel patches for NT os first time .

I didn't. @WildBill is/was a very careful and thorough reverser, who took for himself the mission of porting to 2k all the patches MS was (and remains) denying it. He studied carefully each pach until he fully understood it, then made the port. His ports are safer even than MS's originals. He went even further and reversed kernell32.dll to the point he had a working compilable source he then made public, thus providing a precious source of information regarding what he found out about kernell32.dll to anyone carefull enough to read and meditate about his soure code. And also gave people his free PE tool.
But you're wrong about one detail: @blackwingcat had been extending the 2k kernel some time before @WildBill began to release his own work, and IIRR, the 1st ever to extend the 2k kernel was @OldCigarette  (whom, I'm sad to say, I'm even surer isn't among us anymore). 

[-SnooPY-]

On 3/6/2019 at 6:33 PM, Mathwiz said:

But TLS continues to evolve, so to avoid eventual obsolescence we'd like to get ProxHTTPSProxyMII working with newer Cryptography versions.

This required a newer version of Python, so we've got Python 3.7.1 working on XP; [...]

Cryptography performance does not depend on the version of Python. The only problem is that the new cryptography package (v2.5 - v2.6.1) includes incompatible pre-built binaries (_openssl-cp37-win32.pyd, etc ...), which for some reason is not possible to rebuild with XP support.

Although @hotnuma already pointed to the source code: https://github.com/openssl/openssl/blob/master/crypto/rand/rand_win.c
We are interested in lines from 20 to 40. It is enough not to link bcrypt.lib and the compatibility problem will disappear.

[Mathwiz]

5 hours ago, -SnooPY- said:

Cryptography performance does not depend on the version of Python.

Are you saying that, if you have Win 7 (and therefore bcrypt.dll), the latest Cryptography modules are compatible with Python 3.4? If so, then why are we even bothering with Python 3.7? Seems we're just making things harder for no benefit.

If not, then I don't understand why you said that.

5 hours ago, -SnooPY- said:

@hotnuma already pointed to the source code: https://github.com/openssl/openssl/blob/master/crypto/rand/rand_win.c

Uh, that's source to a module of OpenSSL, not Cryptography. Have you tried to compile OpenSSL on Win XP with, say, MSVC?

Seems to me you need to do that first; then rebuild Cryptography, linking to the OpenSSL .dll's you just built.

[-SnooPY-]

23 hours ago, Mathwiz said:

Are you saying that, if you have Win 7 (and therefore bcrypt.dll), the latest Cryptography modules are compatible with Python 3.4?

I apologize for not being clear. Personally, I only have Win XP, but as far as I understood, @heinoganda successfully tested the latest Cryptography up to 2.6.1 under Win 7 with Python 3.4. In theory, in newer versions of Python (3.5, 3.6 and 3.7) there should also be no problems under Win 7. I just meant that.

23 hours ago, Mathwiz said:

Have you tried to compile OpenSSL on Win XP with, say, MSVC?

IMHO, we already have OpenSSL with XP support in the form of libcrypto-1_1.dll and libssl-1_1.dll files. Unfortunately, I still can not figure out how to build *. pyd files for Python. Cryptohraphy package is not well documented for this case.

Edited March 8, 2019 by -SnooPY-

[Mathwiz]

OK that makes more sense to me now. Thanks for clarifying.

BTW, I checked out some OpenSSL binaries on the Web, and every build of version 1.1.1b I found had a dependency on bcrypt.dll - even the ones billed as "XP compatible." So I suspect those are typically built on Win 7, 8.1, or 10, and the builders are unaware they need to override the definition of WIN32_WINNT to 0x0501 when compiling if they want to retain XP compatibility.

Builds of 1.0.2 and 1.1.0 were generally OK. But those versions are only supported until later this year. We need to see if someone here (maybe @CoRoNe or @roytam1) can rebuild OpenSSL v1.1.1b correctly for XP.

[-SnooPY-]

5 hours ago, Mathwiz said:

So I suspect those are typically built on Win 7, 8.1, or 10, and the builders are unaware they need to override the definition of WIN32_WINNT to 0x0501 when compiling if they want to retain XP compatibility.

Exactly! And the problem with Cryptography in Python is just a consequence of this "inaccuracy".

Edited March 8, 2019 by -SnooPY-

[Reino]

6 hours ago, Mathwiz said:

I checked out some OpenSSL binaries on the Web, and every build of version 1.1.1b I found had a dependency on bcrypt.dll - even the ones billed as "XP compatible." So I suspect those are typically built on Win 7, 8.1, or 10, and the builders are unaware they need to override the definition of WIN32_WINNT to 0x0501 when compiling if they want to retain XP compatibility.

I'm still on WinXP, so I've also compiled openssl-x86-v1.1.1a_legacy.7z on WinXP.

https://github.com/openssl/openssl/blob/master/crypto/rand/rand_win.c#L22:

# if defined(_MSC_VER) && defined(_WIN32_WINNT) && _WIN32_WINNT >= 0x0601

With WIN32_WINNT being 0x0501 this should return false, should it not?

Edited March 8, 2019 by CoRoNe

[Mathwiz]

Excellent work! I just looked with Dependency Walker and it seems like your version of libcrypto-1_1.dll does not have a dependency on bcrypt.dll. So I think you're right and it compiled with the legacy code.

So far the only thing I've found on building Python's Cryptography module is here: https://cryptography.io/en/latest/installation/#building-cryptography-on-windows.

If I read it correctly, one would need your OpenSSL libraries and the OpenSSL include directory, which is part of the source on GitHub. I'm guessing the "2010" and "2015" on that page refer to MSVC versions, and that you used 2010 to compile, so now maybe we can build Cryptography-2.5 or later so it will run on WinXP, at least for Python 3.4.

[-SnooPY-]

7 hours ago, Mathwiz said:

I just looked with Dependency Walker and it seems like your version of libcrypto-1_1.dll does not have a dependency on bcrypt.dll.

You can also check OpenSSL 1.1.1b MinGW build from the Curl for Windows project. The only thing left is to build the latest Cryptography using these DLLs. Does anyone know how to build * .pyd files?

[Reino]

8 hours ago, Mathwiz said:

I'm guessing the "2010" and "2015" on that page refer to MSVC versions, and that you used 2010 to compile...

The compilation of OpenSSL 1.1.1 is part of my FFmpeg buildscript, which uses Cygwin and MinGW, so no MSVC.

I've only visited this thread because you notified me, Mathwiz. What exactly is this Cryptography project? What can it be used for?

[Thomas S.]

6 hours ago, CoRoNe said:

What exactly is this Cryptography project? What can it be used for?

https://pypi.org/project/cryptography/

"Cryptography is a package which provides cryptographic recipes and primitives to Python developers."

Eg. it is used in the project "HTTPSProxy" which provide actual TLS protocols to XP.

Here the discussion about begins:

@heinoganda and I have build versions of the proxy (eg. see my signature).

With cryptography > 2.4.2 we get errors in Python 3.4.4 (the latest for XP), the proxy will not run.

The original python script of the proxy will be found here:

https://prxbx.com/forums/showthread.php?tid=2191&pid=19252#pid19252

Edited March 9, 2019 by Thomas S.

[heinoganda]

Additional Information:

To generate your own cryptography module, the desired module is downloaded from the cryptography project website (https://github.com/pyca/cryptography/releases) (cryptography-2.6.1.zip). Then we need an appropriate OpenSSL release (best from the release infrastructure of the cryptography project Unfortunately, the website is not always available!) and for Python 3.4.4 Windows Native Code Development (VS2010) from the Microsoft Windows SDK (for Windows 7) DVD (GRMSDK_EN_DVD.iso). From Python 3.5 Windows Native Code Development (VS2015).

Further Information https://cryptography.io/en/latest/installation/#building-cryptography-on-windows. I have successfully created the cryptography module 2.6.1 with OpenSSL 1.1.0j (openssl-1.1.0j-2010-x86.zip from the release infrastructure of the cryptography project) for Python 3.4.4. Now the problem is that from OpenSSL 1.1.1a (openssl-1.1.1a-2010-x86.zip) compatibility is no longer given to Windows XP. We need a customized version of openssl-1.1.1b-2010-x86.zip and openssl-1.1.1b-2015-x86.zip that still works on Windows XP.
 

 

Edited March 10, 2019 by heinoganda

[Mathwiz]

On 3/9/2019 at 3:35 AM, -SnooPY- said:

You can also check OpenSSL 1.1.1b MinGW build from the Curl for Windows project. The only thing left is to build the latest Cryptography using these DLLs. Does anyone know how to build * .pyd files?

 

On 3/9/2019 at 4:32 AM, CoRoNe said:

The compilation of OpenSSL 1.1.1 is part of my FFmpeg buildscript, which uses Cygwin and MinGW, so no MSVC.

Both of these appear to be free of bcrypt.dll dependencies, proving one can build OpenSSL 1.1.1 that works on Windows XP.

The build from the Curl for Windows project also includes the /include and /lib subdirectories; I think these are required because the Cryptography module is a static build; i.e., instead of loading the OpenSSL .dll's when called, the OpenSSL code is included when Cryptography itself is built. (IOW, to build Cryptography we don't need the .dll's; we need the same code in lib format.)

However, there is a difference in naming between the Curl for Windows /lib files and the Cryptography project's /lib files. The former have .a extensions while the latter have .lib extensions. I'm guessing this reflects a difference between the MinGW compiler (uses .a) and MSVC (uses .lib). Not sure if the different extensions imply a different format for the lib files.

To be safe I suppose it would be best to rebuild OpenSSL 1.1.1b with MSVC. Looks like a complex build though; you'll need:

Quote

 - Perl. We recommend ActiveState Perl, available from
   https://www.activestate.com/ActivePerl. Another viable alternative
   appears to be Strawberry Perl, http://strawberryperl.com.
   You also need the perl module Text::Template, available on CPAN.
   Please read NOTES.PERL for more information.

 - Microsoft Visual C compiler. Since we can't test them all, there is
   unavoidable uncertainty about which versions are supported. Latest
   version along with couple of previous are certainly supported. On
   the other hand oldest one is known not to work. Everything between
   falls into best-effort category.

 - Netwide Assembler, a.k.a. NASM, available from https://www.nasm.us,
   is required. Note that NASM is the only supported assembler. Even
   though Microsoft provided assembler is NOT supported, contemporary
   64-bit version is exercised through continuous integration of
   VC-WIN64A-masm target.

...

 If you link with static OpenSSL libraries then you're expected to
 additionally link your application with WS2_32.LIB, GDI32.LIB,
 ADVAPI32.LIB, CRYPT32.LIB and USER32.LIB.

I didn't see a 32-bit version of ActiveState Perl on their website, but there is a 32-bit version of Strawberry Perl and of NASM, so it should be possible to build OpenSSL on a 32-bit system with MSVC.

The .LIB files mentioned above are included with VS 2010 (and presumably VS 2015 also). Good luck!

Edited March 11, 2019 by Mathwiz