Ticket #1233: test-prefer-bdists.darcs.patch

1 patch for repository zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk:

Wed Nov  3 14:19:41 MDT 2010  zooko@zooko.com
  * setup: test whether the build system prefers binary dists which are new enough over source dists which are even newer
  ref: #1233

New patches:

[setup: test whether the build system prefers binary dists which are new enough over source dists which are even newer
zooko@zooko.com**20101103201941
 Ignore-this: e06b6cd3cd877458cf47c56805bf4b35
 ref: #1233
] {
addfile ./misc/build_helpers/test-with-fake-dists.py
hunk ./misc/build_helpers/test-with-fake-dists.py 1
+#!/usr/bin/env python
+
+# We put a fake "pycryptopp-0.5.24.egg" package and a fake
+# "pycryptopp-0.5.25.tar.gz" into a directory, but the latter is
+# booby-trapped so it will raise an exception when you try to build
+# it.
+
+# Then we run "python setup.py test -s
+# buildtest.test_with_fake_dist", which imports pycryptopp
+# and fails unless pycryptopp.__version__ == '0.5.24'.
+
+# The goal is to turn red if the build system tries to build the
+# source dist when it could have used the binary dist.
+
+# (Note that for this test to make sense, tahoe-lafs must require a
+# version of pycryptopp which can be satisfied by either 0.5.24 or
+# 0.5.25. At the time of this writing it requires >= 0.5.20 on x86 and
+# >= 0.5.14 on other architectures.)
+
+import StringIO, glob, os, platform, subprocess, sys, tarfile, zipfile
+import pkg_resources
+
+fake_distdir = 'tahoe-deps'
+fake_distname = "pycryptopp"
+fake_sdistversion = "9.9.99"
+fake_bdistversion = "0.5.24"
+sdist_setup = "raise Exception('Aha I caught you trying to build me.')"
+
+testsuite = "buildtest.test_build_with_fake_dist"
+
+dist_dirname = os.path.join(os.getcwd(), fake_distdir)
+
+try:
+    os.makedirs(dist_dirname)
+except OSError:
+    # probably already exists
+    pass
+
+bdist_egg_name = os.path.join(dist_dirname, '%s-%s-py%s.%s-%s.egg' % (fake_distname, fake_bdistversion, platform.python_version_tuple()[0], platform.python_version_tuple()[1], pkg_resources.get_supported_platform()))
+try:
+    bdist_egg = zipfile.ZipFile(bdist_egg_name, 'w')
+    bdist_egg.writestr('pycryptopp/__init__.py', '__version__ = "%s"\n' % (fake_bdistversion,))
+    bdist_egg.close()
+
+    sdist_name = os.path.join(dist_dirname, '%s-%s.tar' % (fake_distname, fake_sdistversion))
+    sdist = tarfile.open(sdist_name, 'w:gz')
+    sdist.errorlevel =2
+    tarinfo = tarfile.TarInfo('setup.py')
+    tarinfo.errorlevel =2
+    tarinfo.size = len(sdist_setup)
+    sdist.addfile(tarinfo, StringIO.StringIO(sdist_setup))
+    sdist.close()
+
+    setuppy="setup.py"
+    os.environ['PATH']=os.path.join(os.getcwd(), 'src')+os.pathsep+os.environ['PATH']
+    eggs = [os.path.realpath(p) for p in glob.glob(os.path.join('..', '*.egg')) if not 'pycryptopp' in p]
+    os.environ['PYTHONPATH']=os.pathsep+os.pathsep.join(eggs)+os.pathsep+os.environ.get('PYTHONPATH','')
+    sys.exit(subprocess.call([sys.executable, setuppy, "-v", "test", "-s", testsuite], env=os.environ))
+finally:
+    os.remove(bdist_egg_name)
+    os.remove(sdist_name)
+    pass
addfile ./src/allmydata/test/test_build_with_fake_dist.py
hunk ./src/allmydata/test/test_build_with_fake_dist.py 1
+#!/usr/bin/env python
+
+import unittest
+
+class T(unittest.TestCase):
+    def test_version(self):
+        import pycryptopp
+        self.failUnlessEqual(pycryptopp.__version__, '0.5.24')
}

Context:

[tests: make test-with-fake-pkg's unacceptably-old fake pycryptopp be 0.5.13 instead of 0.5.17, since 0.5.17 is acceptably new on non-x86 archs
zooko@zooko.com**20101102053357
 Ignore-this: a400f6f73c7574f2b2266547b0e7b051
 Also move the fakepkgs directory to be in misc/build_helpers/ instead of in the base of the source tree.
] 
[test_runner.py: fix test failure in test_the_right_code after applying zooko's change to test-with-fake-pkg.py
david-sarah@jacaranda.org**20101102035905
 Ignore-this: 71b467615ae7dcdfbf84049e60a08933
] 
[setup: FreeStorm's WinXP-x86-py2.6 buildslave has informed us that there is yet a fourth way to spell "i386" in Python
zooko@zooko.com**20101101052142
 Ignore-this: 732892975c19f7fd18caeb071f09ef7
] 
[setup: add platform.machine() to show-tool-versions, because FreeStorm WinXP builder apparently thinks that its platform.machine() is not one of ['i386', 'x86_64', 'amd64', ''], since it is requiring pycryptopp >= 0.5.14 instead of pycryptopp >= 0.5.20
zooko@zooko.com**20101101042721
 Ignore-this: 322a3e5af9785ebca3452f3937ce47a4
] 
[edit docs for English usage, rename "Tahoe" to "Tahoe-LAFS" in docs/configuration.rst, rename a variable in bin/tahoe-script.template for clarity, update François's CREDITS entry
zooko@zooko.com**20101101042602
 Ignore-this: d192c4c9660b1b9173db19d0e533dc22
] 
[allmydata/__init__.py: move the call to require_auto_deps() to the top again, since the [4784] patch turned out not to be the cause of the failure on the 'mm netbsd5' buildslave.
david-sarah@jacaranda.org**20101101003316
 Ignore-this: 879a5188ff2529fb755b6e594c59aeb2
] 
[allmydata/__init__.py: put the _auto_deps.require_auto_deps() call back in exactly the same place it was prior to [4784].
david-sarah@jacaranda.org**20101031174649
 Ignore-this: c5f4fbb2c444d7b05b1d30a199b6178f
] 
[allmydata/__init__.py: suppress a DeprecationWarning for the sha module on importing pycrypto.
david-sarah@jacaranda.org**20101031160636
 Ignore-this: 414d55a0da432cfb0d65329a88e13396
] 
[misc/build_helpers/run_trial.py: fix another typo in last patch.
david-sarah@jacaranda.org**20101031160332
 Ignore-this: 61131c5775a2393b6862f76b7bc222f
] 
[misc/build_helpers/run_trial.py: fix typo in last patch.
david-sarah@jacaranda.org**20101031155215
 Ignore-this: a2fbecf858c0a399e938d4f1ade7329b
] 
[allmydata/__init__.py: call require_auto_deps() after importing nevow and twisted, reverting change in [4784]. Also fix a missing 'warnings.filters.pop()'.
david-sarah@jacaranda.org**20101031153828
 Ignore-this: 8a5cd7798674d56868e9c333a77a4ac2
] 
[misc/build_helpers/run_trial.py: fix false positive on directory check that can occur when running run_trial from test-with-fake-pkg manually.
david-sarah@jacaranda.org**20101031153613
 Ignore-this: 7d4a0758a305cbfdd296570a9c1a88d2
] 
[allmydata.__init__.py: temporary hack to debug failure on midnightmagic's buildslave
david-sarah@jacaranda.org**20101031055003
 Ignore-this: 2ac28b2f19a436a374399b4c59d29cc7
] 
[NEWS: entries for #1190 and #1212, and minor cleanups. refs #1190, #1212
david-sarah@jacaranda.org**20101031051426
 Ignore-this: c318dff69296ae1e1a897752b5221870
] 
[tahoe backup: perform tilde expansion in exclude-from filename (version 2). fixes #1241
david-sarah@jacaranda.org**20101031035231
 Ignore-this: 65e6cd2247dd8d1fc025758d740708c0
] 
[NEWS: add news entry for #1223
Francois Deppierraz <francois@ctrlaltdel.ch>**20101030111130
 Ignore-this: 6b6afd4b0f0527a3c9784c1db95d083
] 
[NEWS: add a NEWS entry about bug #1045
Francois Deppierraz <francois@ctrlaltdel.ch>**20101030101351
 Ignore-this: 7e758afbbd0f1d22a5d0b4fc38661c1d
] 
[setup: run require_auto_deps() before attempting to import any deps in __init__.py
zooko@zooko.com**20101030081035
 Ignore-this: ffcaf2450628543e020e9919e455f691
 For one thing, this makes missing-dependency failures into DistributionNotFound errors instead of ImportErrors, which might be more useful to the user. For another thing, if someone is using distributions that were installed with --multi-version, then they might be not importable until after require_auto_deps() has been run. (The docs claim that this would be the case, but we don't have an example of this happening at this time.)
] 
[setup: show-tool-versions: emit module and __version__ information even when module name != distribution (package) name, and add TwistedCore, TwistedWeb, and TwistedConch
zooko@zooko.com**20101030070233
 Ignore-this: 3df19910090d44502ddeeef5d9c29a7
] 
[misc/build_helpers/test-with-fake-pkg.py: look for eggs in the parent of the src directory. refs #1190
david-sarah@jacaranda.org**20101030034303
 Ignore-this: 4a3cf286272cdb5d06aac15fb5998b33
] 
[scripts/runner.py: fix unused import of allmydata. refs #1190
david-sarah@jacaranda.org**20101030003149
 Ignore-this: b2fc67f6192ea7ccf8a5ad010ce74a64
] 
[scripts/runner.py: remove pkg_resources.require() calls. These are at best redundant because we have already called _auto_deps.require_auto_deps() (from allmydata.__init__) at that point, and they are causing failure of the test-from-prefixdir step on some buildslaves. refs #1190
david-sarah@jacaranda.org**20101029235328
 Ignore-this: e00dee63acc7b76a5755025d75abf524
] 
[misc/build_helpers/run_trial.py: look for zetuptoolz egg in the parent directory, not the cwd of run_trial. refs #1190
david-sarah@jacaranda.org**20101029230329
 Ignore-this: 1596fb8c290d1c706f079701b1857db8
] 
[bundled zetuptoolz: if __main__.__requires__ exists then do not add packages to the working set if they provide an incompatible version of a package. Also put a complete __requires__ listing the transitive closure of dependencies at the beginning of generated scripts, rather than a shallow __requires__ specifying only the application version. refs #1190
david-sarah@jacaranda.org**20101029223111
 Ignore-this: a95f1967884340e53bf3adf90db40cfc
] 
[setup.py, misc/build_helpers/run_trial.py: use undocumented __requires__ variable to cause setuptools/zetuptoolz to put the correct versions of dependencies on sys.path. Also ensure that run_trial adds the bundled zetuptoolz egg at the start of sys.path if present. Make the source directory comparison work correctly for the test-with-fake-pkg build step. refs #1190
david-sarah@jacaranda.org**20101029222825
 Ignore-this: 8b09366eb6ce3d55c7db5239077a0fac
] 
[test_runner.py: fix error in BinTahoe.test_version_no_noise introduced by last patch. refs #1235
david-sarah@jacaranda.org**20101029221123
 Ignore-this: 4bf21ea34768e8e6adf104e56f939fd0
] 
[test_runner.py: also allow 'from pkg_resources import load_entry_point' as noise. refs #1235.
david-sarah@jacaranda.org**20101029204246
 Ignore-this: a47440aa2cdd29ce55ac7c6c7f4bcaf2
] 
[test_runner.py: if the only noise is 'UserWarning: Unbuilt egg for setuptools', skip instead of failing the no_noise tests. This version avoids 'any' to be compatible with Python < 2.5. refs #1235.
david-sarah@jacaranda.org**20101029191804
 Ignore-this: 83ca1543fc9673e664a8eeefe1eba429
] 
[NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232
david-sarah@jacaranda.org**20101028180046
 Ignore-this: 1c3eef3cd353b06b6ee00ce87c5ef59a
] 
[windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.
david-sarah@jacaranda.org**20101027021636
 Ignore-this: fbd99e0d22493974696d37925d97c7d6
] 
[scripts/tahoe_backup.py: oops, fix missing import, thanks pyflakes
Brian Warner <warner@lothar.com>**20101029094223
 Ignore-this: 285c35af824935641a5be35c008b080c
 
 test_cli.py: hush minor pyflakes complaint
] 
[mutable/servermap.py: update comment. Closes #1231.
Brian Warner <warner@lothar.com>**20101029091424
 Ignore-this: 80bf854123fc254e097a81b82bdf4990
] 
[tahoe_cp.py: Don't call urllib.quote with an Unicode argument, fix #1224
Brian Warner <warner@lothar.com>**20101029084520
 Ignore-this: 5524722d5e5babbb73ca0969d54967f6
 tahoe_backup.py: Fix another (potential) occurrence of calling urllib.quote()
 with an Unicode parameter
] 
[fix #1223, crash+inefficiency during repair due to read overrun
Brian Warner <warner@lothar.com>**20101029082036
 Ignore-this: e6aa0295ad254544da3b5cc41b33d862
 
 * repairer (really the uploader) reads beyond end of input file (Uploadable)
 * new-downloader does not tolerate overreads
 * uploader does lots of tiny reads (inefficient)
 
 This fixes the last two. The uploader still does a single overread at the end
 of the input file, but now that's ok so we can leave it in place. The
 uploader now expects the Uploadable to behave like a normal disk
 file (reading beyond EOF will return less data than was asked for), and now
 the new-downloadable behaves that way.
] 
[make ResponseCache smarter to avoid memory leaks: don't record timestamps, use DataSpans to merge entries, and clear the cache when we see a new seqnum. refs #1045, #1229
david-sarah@jacaranda.org**20101027043302
 Ignore-this: 88fd6fba7f35a2f8af1693b92718f5f3
] 
[add misc/build_helpers/test-with-fake-pkg.py. refs #1190
david-sarah@jacaranda.org**20101029025150
 Ignore-this: 995f220962708f1bad83092161130f67
] 
[startstop_node.py: pyflakes import fix. refs #1190
david-sarah@jacaranda.org**20101028014805
 Ignore-this: 369ef5022c8ee5a0d8341af01553bcef
] 
['tahoe start': use import+call rather than find+spawn
"Brian Warner <warner@lothar.com>"**20101027061342
 
 This removes the need to use a locally-built (dependency) bin/twistd, and
 removes a big chunk of behavior differences between unix and windows. It
 also happens to resolve the "client node probably started" uncertainty.
 Might help with #1190, #602, and #71.
] 
[docs/known_issues.rst: Add section on traffic analysis. Fix URL for current version of file.
david-sarah@jacaranda.org**20101024234259
 Ignore-this: f3416e79d3bb833f5118da23e85723ad
] 
[test_mutable.py: add test for ResponseCache memory leak. refs #1045, #1129
david-sarah@jacaranda.org**20101024193409
 Ignore-this: 3aee7f0677956cc6deaccb4d5b8e415f
] 
[test_encodingutil.py: test_argv_to_unicode modified the wrong encoding variable. fixes #1214
david-sarah@jacaranda.org**20101023035810
 Ignore-this: e5f1f849931b96939facc53d93ff61c5
] 
[docs/running.html: fix missing end-quote, and change frontends/ doc references to .rst.
david-sarah@jacaranda.org**20101024171500
 Ignore-this: 47c645a6595e1790b1d1adfa71af0e1d
] 
[docs/running.html: 'tahoe create-client' now creates a node with storage disabled. Also change configuration.txt references to configuration.rst.
david-sarah@jacaranda.org**20101024170431
 Ignore-this: e5b048055494ba3505bb8a506610681c
] 
[docs: fix tab-vs-spaces, make some CLI examples <tt>/"literal", wrap some to
Brian Warner <warner@lothar.com>**20101015060606
 Ignore-this: eae08bdf0afb19a2fbf41c31e70a8122
 80-cols, remove spurious whitespace. Add rst2html.py rule to Makefile.
] 
[doc: add explanation of the motivation for the surprising and awkward API to erasure coding
zooko@zooko.com**20101015060202
 Ignore-this: 428913ff6e1bf5b393deffb1f20b949b
] 
[setup: catch and log ValueError from locale.getdefaultlocale() in show-tool-versions.py
zooko@zooko.com**20101015054440
 Ignore-this: 827d91490562c32ed7cf6526dfded773
 I got a bug report from Mathias Baert showing that locale.getdefaultlocale() raises an exception on his Mac OS X system. Heh.
] 
[docs: update how-to-make-a-release doc with a few tweaks from the 1.8.0 process
zooko@zooko.com**20101015054413
 Ignore-this: ca5e9478531a3393792ae283239549dd
] 
[docs: update NEWS ref: #1216
zooko@zooko.com**20101015053719
 Ignore-this: 2e0b92e4145d667cdf075e64b7965530
] 
[docs: add Peter Secor, Shawn Willden, and Terrell Russell as signatories to docs/backdoors.rst
zooko@zooko.com**20101015053242
 Ignore-this: c77adf819d664f673e17c4aaeb353f33
] 
[docs: convert all .txt docs to .rst thanks to Ravi Pinjala
zooko@zooko.com**20101015052913
 Ignore-this: 178a5122423189ecfc45b142314a78ec
 fixes #1225
] 
[docs: add statement on our refusal to insert backdoors
zooko@zooko.com**20101006051147
 Ignore-this: 644d308319a7b80c4434bdff9760404a
] 
[setup: add --multi-version to the "setup.py develop" command-line
zooko@zooko.com**20101005182350
 Ignore-this: 709155cc21caff29826b8d41a8c8d63d
 fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.)
] 
[setup: use execfile to access _auto_deps.py in its proper location of src/allmydata/ instead of copying it into place when setup.py is executed
zooko@zooko.com**20100906055714
 Ignore-this: c179b42672d775580afad40121f86812
] 
[trivial: M-x whitespace-cleanup
zooko@zooko.com**20100903144712
 Ignore-this: 1bb764d11ac69b4a35ea091cfb13158a
] 
[minor: remove unused interface declaration, change allmydata.org to tahoe-lafs.org in email address, fix wording in relnotes.txt
zooko@zooko.com**20100930153708
 Ignore-this: a452969228afed2774de375e29fa3048
] 
[immutable/repairer.py: don't use the default happiness setting when repairing
Kevan Carstensen <kevan@isnotajoke.com>**20100927200102
 Ignore-this: bd704d9744b970849da8d46a16b8089a
] 
[NEWS: note dependency updates to pycryptopp and pycrypto.
david-sarah@jacaranda.org**20100924191207
 Ignore-this: eeaf5c9c9104f24c450c2ec4482ac1ee
] 
[TAG allmydata-tahoe-1.8.0
zooko@zooko.com**20100924021631
 Ignore-this: 494ca0a885c5e20c883845fc53e7ab5d
] 
Patch bundle hash:
b95082dd2453053a47fd70b9acf0913c9755321e