better detection and diagnostics about wrong-code-generation compile errors

[davidsarah]

When using the prebuilt pycryptopp-0.5.29-py2.7-win32.egg hosted at tahoe-lafs.org, the start_up_self_test for SHA-256 would hang at the first call to digest() at line 35 of sha256.py. Rebuilding the egg using Visual Studio 2008 Express (with Windows updates applied) caused this problem to go away. The new egg has now been copied to ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs/pycryptopp-0.5.29-py2.7-win32.egg , and the broken one is at ​http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-eggs/pycryptopp-0.5.29-py2.7-win32.egg.broken .

I don't know whether any of the other pycryptopp eggs are similarly broken.

Note that the symptoms are very similar to the binutils-related bug ​http://tahoe-lafs.org/trac/tahoe-lafs/ticket/853 (caused by ​https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/461303 ). It is possible that the broken pycryptopp-0.5.29-py2.7-win32.egg was built with a broken version of gas, but I don't know how to confirm that.

If there were pycryptopp buildslaves building the eggs for Windows, they would automatically run the unit tests, which would have found this build problem. Building the eggs manually is very error-prone, and information about the build environment is lost. (Four buildslaves are needed: Python 2.6 and 2.7 for 32-bit and 64-bit Windows.)

[davidsarah]

Currently there are three Windows pycryptopp buildslaves: "FreeStorm windows-mingw32-py26", "FreeStorm Win7-amd64-mingw py2.6", and "Dcoder Win7-64 py2.6".

The amd64-mingw one does not work, because of problems using the 64-bit version of mingw to compile Python extension modules.

We should probably consistently publish either eggs built with mingw or with Visual Studio, not a combination of both.

[davidsarah]

Correction: start_up_self_test was probably called from the SHA256 constructor (on the same line of sha256.py) rather than the first call to digest().

[davidsarah]

I mean, the hang was probably in native code called from the SHA256 constructor. (start_up_self_test is the enclosing Python function. I didn't mean that it was called recursively, that was just a mistake.)

[zooko]

So possible improvements that we could make include:

1. Ask people who build binary eggs for us to run the unit tests from the resulting egg.

2. Have the build process (setup.py) embed into the resulting package the names and version numbers of the tools used to build it (for better diagnositcs).

3. Have "python setup.py bdist_egg" run tests first (after building) and error out if tests fail.

Option 3 is sort of like ​http://tahoe-lafs.org/trac/tahoe-lafs/ticket/853#comment:20 . At the time I thought it would be too clever, but I don't feel that way if we treat it as a fail-stop—"run test and abort with a clear error message or at least a seg fault"—instead of as an auto-work-round "run test and if it shows that your compiler is buggy then disable assembly optimizations and try again".