#145 closed defect (duplicate)

"make test" tests the installed version of allmydata, not the local sandbox version of allmydata

Reported by: zooko Owned by: zooko
Priority: major Milestone: soon
Component: packaging Version: 0.6.1
Keywords: test-needed docs-needed Cc:
Launchpad Bug:

Description

It seems like .pth paths get searched earlier than PYTHONPATH, so if you install allmydata and then change something in your local sandbox and then run "make test", it will test the installed version, not the local one that you changed.

Change History (29)

comment:1 Changed at 2007-09-28T02:35:10Z by warner

  • Component changed from unknown to code
  • Owner changed from nobody to somebody

eek. I thought that .pth files had path-manipulation logic in them to make sure that everything which gets added is really inserted into the middle of sys.path, in the same place that the .pth file was found.

Hmm. I went to confirm this, and realized that I can't seem to find any .pth files on my system that contain code, and I can't seem to find a reference to '.pth' in the site.py files there.

comment:2 Changed at 2007-10-01T20:04:57Z by zooko

  • Owner changed from somebody to zooko
  • Status changed from new to assigned

comment:3 Changed at 2007-10-13T06:40:31Z by zooko

This one bit me again, as I sleepily couldn't figure out why my new unit test wasn't failing as it should. When I deleted the installed version of allmydata, this was fixed.

One workaround to this which might not be a terrible idea in any case is to exclude test files from the installed files.

comment:4 Changed at 2007-10-13T23:11:54Z by warner

I'd like to understand the reasons first.. what was the PYTHONPATH used for that test?

I'm also inclined to leave the test sub-package installed, since that allows users who have just installed tahoe (perhaps from some binary installer) to run the unit tests on their own. FWIW, Twisted does the same thing, and following their lead I did the same for Buildbot and Foolscap.

comment:5 Changed at 2007-10-15T13:54:04Z by zooko

When tahoe isn't installed, then make test starts with:

PYTHONPATH="/Users/wonwinmcbrootles/playground/allmydata/tahoe/src::" \
 python -u "/usr/local/bin/trial" --rterrors   allmydata.test
Running 233 tests.

I'll investigate more later...

comment:6 Changed at 2007-10-15T16:33:41Z by zooko

  • Owner changed from zooko to warner
  • Status changed from assigned to new

Okay here is the difference between sys.path when running unit tests when tahoe is installed vs. when it is not installed:

-    test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/allmydata_tahoe-0.6.0_76-py2.5-macosx-10.3-i386.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages']
+    test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages']

So all of the installed .eggs from the Python site-packages dir, including the allmydata-tahoe .egg, are earlier on the path than $PWD/src is.

comment:7 Changed at 2007-10-15T20:26:51Z by zooko

  • Milestone changed from 0.6.1 to 0.7.0

Bumping to v0.7.

comment:8 Changed at 2007-10-18T23:28:06Z by zooko

  • Milestone changed from 0.7.0 to 0.6.2

comment:9 Changed at 2007-10-24T00:53:34Z by warner

  • Priority changed from major to critical

this is currently giving us false-positive test results on the Solaris buildslave, and possibly others, so I'm raising the priority.

One way to lower the priority would be to modify the test process and buildbot config to emit the version number of the code actually being tested (as extracted from the _trial_temp/twistd.log file) in the test step, so we could spot the discrepancy. At the moment, the buildbot is silently testing the wrong versions on platforms where tahoe was installed into a normal system path.

comment:10 Changed at 2007-10-24T01:27:06Z by warner

from the following mailing list articles:

http://www.eby-sarna.com/pipermail/peak/2006-june/002582.html http://mail.python.org/pipermail/distutils-sig/2006-July/006492.html

it looks like using eggs always results in the following import order:

  • eggs on PYTHONPATH
  • eggs in site-packages
  • non-eggs on PYTHONPATH
  • non-eggs in site-packages

and for us, the tahoe "egg in site-packages" is overriding the code-under-test "non-egg on PYTHONPATH".

It appears that the recommended solution might be to use "setup.py develop", to create an .egg (really a directory, rather than a .zip file) that contains symlinks into our source tree. Then put that .egg in a local directory on PYTHONPATH.

comment:11 Changed at 2007-11-01T17:09:07Z by zooko

  • Milestone changed from 0.6.2 to 0.7.1
  • Priority changed from critical to major
  • Version changed from 0.5.1 to 0.6.1

We're focussing on an imminent v0.7.0 which hopefully has Small Distributed Mutable Files and also a fix for bad SHA-256. So I'm bumping less urgent tickets to v0.7.1.

comment:12 Changed at 2007-11-13T17:48:26Z by zooko

I uninstalled the allmydata-tahoe package from the system on nooxie, so now the buildslave on nooxie is testing its local checkout of the trunk source.

So while this is still a bug, it is no longer the cause of any build failures on nooxie.

comment:13 Changed at 2007-11-13T19:41:52Z by zooko

Brian, do I understand correctly that the fix is for "make test" to do "./setup.py develop" to produce an egg which symlinks into the directory and put that egg on the PYTHONPATH?

comment:14 Changed at 2007-11-13T19:42:00Z by zooko

  • Priority changed from major to minor

comment:15 Changed at 2008-01-23T02:47:59Z by zooko

  • Milestone changed from 0.7.1 to undecided

comment:16 Changed at 2008-02-14T04:12:49Z by zooko

  • Owner changed from warner to zooko
  • Status changed from new to assigned

Nowadays make build runs ./setup.py develop, so this might fix this bug. I haven't tested it yet, though.

comment:17 Changed at 2008-03-04T22:29:49Z by zooko

  • Resolution set to fixed
  • Status changed from assigned to closed

Yes, this is fixed.

comment:18 Changed at 2008-03-13T17:10:50Z by zooko

  • Milestone changed from undecided to 0.9.0 (Allmydata 3.0 final)

comment:19 Changed at 2008-03-16T08:01:28Z by zooko

  • Resolution fixed deleted
  • Status changed from closed to reopened

Hm. This is apparently not fixed. If you easy_install tahoe then make test tests that installed version instead of the one in ./support.

comment:20 Changed at 2008-05-29T22:31:39Z by warner

  • Milestone changed from 1.1.0 to 1.2.0

comment:21 Changed at 2008-09-22T20:12:08Z by zooko

  • Component changed from code to packaging

comment:22 Changed at 2008-10-22T01:32:41Z by zooko

  • Resolution set to fixed
  • Status changed from reopened to closed

No, I checked and this seems to be fixed. If anyone else can reproduce the problem -- make test testing some other copy of the allmydata-tahoe source code instead of the source code in the current directory -- please let us know.

Also, I guess, if ./setup.py trial tests any allmydata package other than the one in the current directory, that too would mean this ticket should be re-opened.

comment:23 Changed at 2010-07-27T06:07:38Z by davidsarah

  • Keywords test-needed added
  • Milestone changed from 1.5.0 to soon
  • Priority changed from minor to major
  • Resolution fixed deleted
  • Status changed from closed to reopened

See #1137 for a reoccurrence of this problem, in the test-from-egg and test-from-prefixdir buildbot steps.

Why do we think it was fixed for setup.py trial? I see no test that would fail if we were testing the wrong code. (test_path in test_runner.py checks that we run the same code by invoking bin/tahoe that we are currently testing, but if bin/tahoe and setup.py trial were running the same installed version -- which is entirely plausible because they both work by setting PYTHONPATH -- then we wouldn't detect that.)

Last edited at 2010-07-27T06:10:00Z by davidsarah (previous) (diff)

comment:24 follow-up: Changed at 2010-07-27T06:55:39Z by zooko

Hm, test_path in test_runner.py detects whether starting from the allmydata directory that contains the allmydata/__init__.py file and going ../../bin/tahoe finds a working tahoe executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:

 Zooko-Ofsimplegeos-MacBook-Pro:~$ python -i
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import allmydata
>>> import os
>>> os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))), 'bin', 'tahoe')
'/Library/Python/2.6/site-packages/bin/tahoe'
>>> ^D
 Zooko-Ofsimplegeos-MacBook-Pro:~$ ls -al /Library/Python/2.6/site-packages/bin/tahoe
ls: /Library/Python/2.6/site-packages/bin/tahoe: No such file or directory
 Zooko-Ofsimplegeos-MacBook-Pro:~$ python -c 'import allmydata;print allmydata.__file__'
/Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg/allmydata/__init__.pyc

Ah, but in this case the test is skipped:

HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path
allmydata.test.test_runner
  TheRightCode
    test_path ...                                                     [SKIPPED]

===============================================================================
[SKIPPED]
The bin/tahoe script isn't to be found in the expected location (/Library/Python/2.6/site-packages/bin/tahoe), and I don't want to test a 'tahoe' executable that I find somewhere else, in case it isn't the right executable for this version of Tahoe. Perhaps running 'setup.py build' again will help.

allmydata.test.test_runner.TheRightCode.test_path
-------------------------------------------------------------------------------
Ran 1 tests in 0.011s

PASSED (skips=1)

So this test does fail in (some) cases that it is running a test_runner.py from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

comment:25 in reply to: ↑ 24 ; follow-up: Changed at 2010-07-29T03:31:23Z by davidsarah

Replying to zooko:

Hm, test_path in test_runner.py detects whether starting from the allmydata directory that contains the allmydata/__init__.py file and going ../../bin/tahoe finds a working tahoe executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:

[...]

Ah, but in this case the test is skipped:

HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path
allmydata.test.test_runner
  TheRightCode
    test_path ...                                                     [SKIPPED]

Actually the test will always be skipped rather than failing because of this. It is skipped in the case in #1137, for example.

So this test does fail in (some) cases that it is running a test_runner.py from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, ...

I think running the tests on an installed version (as opposed to a version that is built at a prefixdir) isn't currently a supported thing to do; you have to be in a directory that contains at least a setuptools-*.egg, and presumably that also looks like a Tahoe-LAFS source distribution in other ways (the missing setuptools-*.egg is as far as I got). That it passes when you import an installed version instead of the one in src/allmydata is an accident; I'll open another ticket to support that case.

and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

The test_the_right_code test that I added on the ticket1074 branch should fail in this case, but an additional problem is that the version we're incorrectly testing might not have that test (or, we might fix a bug in that the test in future, and be confused by its difference in behaviour in the version we're actually testing).

Making a change in setuptools_trial would have the same problem because we might be using an installed version of that. Making a change in build_helpers/run_trial.py would work, though. I'm trying that approach now.

comment:26 in reply to: ↑ 25 Changed at 2010-07-29T05:29:07Z by zooko

Hm, let's list the supported ways to run tests:

  1. cd $SOURCEDIR && python setup.py test
  2. cd $SOURCEDIR && trial allmydata.test
  3. mkdir tempempty && cd tempempty && trial allmydata.test # should run unit tests of the installed source code, might skip some tests -- test_runner -- that depend on the test code figuring our where the 'tahoe' executable lives
  4. mkdir -p egginstalldir && PYTHONPATH=egginstalldir:${PYTHONPATH} easy_install -d egginstalldir dist/*.egg && cd egginstalldir && PYTHONPATH=.:${PYTHONPATH} trial allmydata.test # should run unit tests of the code in this egginstalldir including test_runner. This is what install-to-egg and test-from-egg on the buildbot is trying to do.
  5. python setup.py install --single-version-externally-managed --record=/dev/null --prefix=prefixinstalldir && cd prefixinstalldir && PYTHONPATH=lib/python2.6/site-packages:${PYTHONPATH} PATH=bin:${PATH} trial allmydata.test # should run unit tests of the code in this prefixinstalldir including test_runner. This is what install-to-prefixdir and test-from-prefixdir on the buildbot is trying to do

Is that it?

Note: I just tried each of these manually and here are my results:

  1. works
  2. does not work; It runs the unit tests of the installed version in /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg instead of the current source tree, even if I set the PYTHONPATH before running trial.
  3. works
  4. works
  5. does not work; It runs the unit tests of the installed version in /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg instead of the current source tree, even if I set the PYTHONPATH before running trial.

comment:27 Changed at 2011-01-22T03:13:23Z by davidsarah

Note that as part of #1296, all of the supported ways to run tests (including the Makefile targets) now directly or indirectly invoke bin/tahoe debug trial. The ways to run tests described in comment:26 that didn't work, still don't work, but they are no longer used on the buildbots and I don't think they should be considered supported.

It is still possible for --single-version-externally-managed installs (and other installs that copy code to site directories, such as the Twisted installers for Windows) to mess things up, although we try very hard to detect such cases. The only such cases I'm aware of are caused by #1258. Can we consider this ticket to be a duplicate of that one?

comment:28 follow-up: Changed at 2011-01-22T05:27:39Z by zooko

  • Keywords docs-needed added

Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named docs/tests.rst?

comment:29 in reply to: ↑ 28 Changed at 2011-07-23T00:32:49Z by davidsarah

  • Resolution set to duplicate
  • Status changed from reopened to closed

Replying to zooko:

Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named docs/tests.rst?

Filed as #1439. I haven't seen any other cases than #1258 where we test the wrong code, so I'm marking this as a duplicate.

Note: See TracTickets for help on using tickets.