Decide where "packaging tests" should live.

[nejucomo]

Task: Define a policy about where packaging tests live, where buildbot configuration lives, then specify that policy on the wiki.

Details:

We'd benefit from "packaging tests" which are different from unittests. The latter target a prepared installation or repository checkout, whereas packaging tests have these distinctions:

Inputs: Packaging tests may target as input:

• build tests - a fresh repository checkout in order to test creation of distribution formats -- e.g. sdist, linux distro packages, sumo tarballs, etc...
• installation tests - a locally prepared distribution format -- e.g. sdist, .deb, etc...
• distribution tests - the network -- e.g. a PyPI package name and http(s) connections to python packaging mirrors; a debian package name and connections to debian repositories; etc...

Staging or Production: We might like to run the same tests against unreleased versions and distinctly against "the live internet".

• Staging tests might rely on a local PyPI mirror to ensure that if we were to distribute sdists to the real PyPI, then users will probably successfully be able to install them.
• Production tests rely on the actual distribution that live users use. Failures of these tests let us know when our released packages are unavailable to users, and can help identify why.

Brittleness: sensitivity to transient conditions:

• Packaging tests start from either a repository clone or an sdist and should behave fairly deterministically without networking.
• Installation tests start from local packaging files, and should behave deterministically, barring system state issues.
• Distributions tests necessarily use the internet and will give false positives for transient errors. (Note: Errors due to transient outages are important information! Suppose tahoe-lafs install relies on some.third-party.package.host.example.com and that site goes down. We want to know that people are now unable to install tahoe-lafs through specific channels.)

We may want different packaging tests to live in different places depending on these axes. Some of these tests may not require buildbot, and/or may not be developed or maintained by us at all! For example, tests which verify live PyPI packages may be run by PyPI itself. We should still account for this in an "official policy".

We probably want test infrastructure operators to have some flexibility to only run some categories of tests. For example, a debian test box would not want to run windows installation tests, or some operators may prefer to limit networking and only run non-distribution tests.

[nejucomo]

Note: We might have some tests that are abstracted over the target package. For example a distribution test which takes as input a PyPI package named $PKG:

1. Create a new pristine virtualenv; activate it.
2. $ pip install $pkg
3. Fail if the installation fails.
4. Fail if http connections were attempted without ssl.
5. Run the installed packages unit tests; fail if they fail.
6. Otherwise, pass.

If this were adopted as a packaging goal/specification, we could apply it to allmydata-tahoe, zfec, pyutil, pycryptopp, etc...

[nejucomo]

Zooko points out that if the tests live in the main tahoe-lafs repository, then when they are invoked they may incorrectly import the local unpackaged code, or make other dependency mistakes.

[daira]

I'm strongly in favour of having these tests in a separate repo on github (and moving any existing such tests into that repo).

[daira]

​https://github.com/tahoe-lafs/packaging-tests ?

[zooko]

Replying to daira:

​https://github.com/tahoe-lafs/packaging-tests ?

Yes, please!

As far as I know, only Brian has the authority to create new repos under ​https://github.com/tahoe-lafs.

(Well, only Brian, all the people who have root at github.com, and DigiCert, and all of the root CAs, and people who have owned root CAs, and the makers of web browsers. I actually hope to fix this before colonizing outer space.)

[zooko]

Replying to daira:

I'm strongly in favour of having these tests in a separate repo on github (and moving any existing such tests into that repo).

Any existing such tests:

• ​TestOldDep
• ​TestAlreadyHaveDep
• ​InstallToEgg
• ​InstallToPrefixDir
• ​TestFromEggTrial
• ​TestFromEgg
• ​TestFromPrefixDirPyunit
• ​TestFromPrefixDirRunTrial

[zooko]

Brian: would you please create a repo named ​https://github.com/tahoe-lafs/packaging-tests ?

[daira]

Replying to zooko:

[...] (Well, only Brian, all the people who have root at github.com, and DigiCert, and all of the root CAs, and people who have owned root CAs, and the makers of web browsers. I actually hope to fix this before colonizing outer space.)

Hmm, I'm not sure that should be a blocker for our interstellar exploration plans ;-)

[zooko]

Oh look there was an old duplicate of this: #1248, which has some relevant detail in it. Please check it out. I'm closing #1248 as a duplicate of this.

[warner]

I'm -0. I'll create that repo if you want, but first I'd like to push back on the notion that these tests ought to live outside of the main source tree.

Zooko points out that if the tests live in the main tahoe-lafs repository, then when they are invoked they may incorrectly import the local unpackaged code, or make other dependency mistakes.

I think it's easy to avoid that. Our bin/tahoe looks in its parent directory for the marker file that indicates it should include ../src and ../support, etc. But if we unpack the sdist into a subdirectory, and run the new bin/tahoe, it won't look in the original tree for anything.

(One counter-example would be if something in the build process wants to know if it's in a git checkout, for which git walks all the way to the filesystem root looking for a .git directory. I don't think that's a big deal.)

More repositories means more places for contributors to look, more divergence between code and tests. Also it raises the buildbot question of which repo's changes should trigger the tests being run (presumably you'd want to trigger on changes in either source-repo or tests-repo, and that's not trivial with buildbot).

My weak suggestion would be to put all of these "packaging tests" into a subdirectory of the main tree, and have the buildbot (or whatever) check out trunk, then run one of them. The packaging tests should not feel obligated to use the rest of the source code for anything, but if the test happens to need a trunk checkout, one is nearby. I could be talked out of this, though.

[zooko]

Well, I still feel like it is better to maintain the packaging tests and the Tahoe-LAFS code separately. I don't foresee it being a problem that there is "divergence" between the packaging tests and the Tahoe-LAFS code. Those two things probably oughtn't be changed in tight synchronization with each other anyway!

I've had problems in the past with packaging code accidentally importing the wrong version of a package, and it was very confusing, so I feel nervous about the packaging tests coming with a copy of the code-under-test in a sibling directory. I think it could cause trouble.

I think it's easy to avoid that. Our bin/tahoe looks in its parent directory for the marker file that indicates it should include ../src and ../support, etc. But if we unpack the sdist into a subdirectory, and run the new bin/tahoe, it won't look in the original tree for anything.

I don't think this is easy! I find this stuff very confusing. You can't rely on setting PYTHONPATH, by the time you can edit sys.path it is sometimes too late, you can't rely on editing sys.modules (although I occasionally have to do that). Having your Python code crawl around in your file system and look for magic files in sibling directories, like we do in bin/tahoe is icky and non-standard, even though it has worked so far bin/tahoe's purposes.

In short, if I want some code (the packaging tests) to have control over what specific copy of some other code (the code-under-test) it starts with, then my preferred way of achieving that is:

1. The test code starts with no extant copies of any version of the other code present on its filesystem.

2. The test code fetches a specifically chosen version, by tarball-download of a specific URL, or by git fetch of a specific branch, tag, or commit-hash.

[warner]

Replying to zooko:

I'm closing #1248 as a duplicate of this.

Ok, but then let's please have "move test code out of buildbot's master.cfg and into some repo" be part of this (#2049) ticket. That was the point of #1248, and it's not clear thaht it was included in the remit for #2049.

[warner]

Eh, ok. Repo created at ​https://github.com/tahoe-lafs/packaging-tests . I+zooko+daira have commit.

[nejucomo]

We chatted about next steps in a weekly dev chat just now. I propose the next steps are:

1. ☑ Review and merge ​https://github.com/tahoe-lafs/buildbot-config-tahoe/pulls
2. ☐ Create a local master/slave setup to test changes.
3. ☐ Create a test runner framework in ​https://github.com/tahoe-lafs/packaging-tests
4. ☐ Ensure that the new test framework executes the same tests with the same output reported as the existing production buildbot deployment.
5. ☐ Swap in this new configuration into the existing build-bot network (-and carefully consider contingency downgrade)
6. ☐ Verify that the new buildbot operation is functionally equivalent to the current operation
7. ☐ Close this ticket as resolved

After that, we can iterate on packaging-tests to try to nail down installation issues, using separate tickets.

Edit: Added checkboxes for the tasks and marked the first as complete.

Edit 2014-09-25: Changed a step from " Move the Tahoe ./misc/build_helpers tests into packaging-tests" to be about non-regression criterion.

Edit 2014-09-25: Inserted a new step: "Create a local master/slave setup to test changes." See comment 25.

[zooko]

We've done step 1 from comment:15.

[nejucomo]

Note: I removed the checkboxes in the quotes to avoid state confusion. (I intended to update comment 15 to reflect new changes in addition to adding comments.)

Replying to nejucomo:

We chatted about next steps in a weekly dev chat just now. I propose the next steps are:

1. Review and merge ​https://github.com/tahoe-lafs/buildbot-config-tahoe/pulls
2. Create a test runner framework in ​https://github.com/tahoe-lafs/packaging-tests

I propose we use twisted trial as "the framework" and run tests with trial $OPTIONS packaging_tests.

3. Move the Tahoe ./misc/build_helpers tests into packaging-tests

Oh, I should look at these before selecting the framework. Hopefully these are easy to hook into trial tests.

4. Swap in this new configuration into the existing build-bot network (-and carefully consider contingency downgrade)
5. Verify that the new buildbot operation is functionally equivalent to the current operation
6. Close this ticket as resolved

After that, we can iterate on packaging-tests to try to nail down installation issues, using separate tickets.

Edit: Added checkboxes for the tasks and marked the first as complete.

So right now I'm going to examine ./misc/build_helpers and determine if they'd fit into a trial design.

[Zancas]

Should nejucomo be given commit access to:

​https://github.com/tahoe-lafs/packaging-tests

and/or:

​https://github.com/tahoe-lafs/buildbot-config-tahoe

?

[daira]

Probably. I don't have admin access to those either.

[Zancas]

Replying to nejucomo:

Task: Define a policy about where packaging tests live, where buildbot configuration lives, then specify that policy on the wiki.

Details:

We'd benefit from "packaging tests" which are different from unittests. The latter target a prepared installation or repository checkout, whereas packaging tests have these distinctions:

Inputs: Packaging tests may target as input:

• build tests - a fresh repository checkout in order to test creation of distribution formats -- e.g. sdist, linux distro packages, sumo tarballs, etc...
• installation tests - a locally prepared distribution format -- e.g. sdist, .deb, etc...
• distribution tests - the network -- e.g. a PyPI package name and http(s) connections to python packaging mirrors; a debian package name and connections to debian repositories; etc...

Staging or Production: We might like to run the same tests against unreleased versions and distinctly against "the live internet".

• Staging tests might rely on a local PyPI mirror to ensure that if we were to distribute sdists to the real PyPI, then users will probably successfully be able to install them.
• Production tests rely on the actual distribution that live users use. Failures of these tests let us know when our released packages are unavailable to users, and can help identify why.

Brittleness: sensitivity to transient conditions:

• Packaging tests start from either a repository clone or an sdist and should behave fairly deterministically without networking.
• Installation tests start from local packaging files, and should behave deterministically, barring system state issues.
• Distributions tests necessarily use the internet and will give false positives for transient errors. (Note: Errors due to transient outages are important information! Suppose tahoe-lafs install relies on some.third-party.package.host.example.com and that site goes down. We want to know that people are now unable to install tahoe-lafs through specific channels.)

We may want different packaging tests to live in different places depending on these axes. Some of these tests may not require buildbot, and/or may not be developed or maintained by us at all! For example, tests which verify live PyPI packages may be run by PyPI itself. We should still account for this in an "official policy".

We probably want test infrastructure operators to have some flexibility to only run some categories of tests. For example, a debian test box would not want to run windows installation tests, or some operators may prefer to limit networking and only run non-distribution tests.

Here're some tweaks to the test type taxonomy outlined above.

1. I think the name "network distribution tests" is easier to understand, perhaps because it's more specific, than "distribution tests". I believe this because when I added the "network " before instances of "distribution test" I found the text easier to understand.

2. If I Understand Correctly (IIUC) the word "Packaging" which is the first word after the first bullet point in the "Brittleness" section is a substitution error, and should be the word "Build".

3. It seems to me that "Brittleness" is a criteria, like "Inputs", by which one can categorize test "packaging test" types, whereas "Staging or Production" is a discriminator that applies more agnostically to all tests. Therefore I propose that the description will have a more intuitive flow if the order of the sections: "Brittleness" and "Staging or Production" are swapped. Also it might be nice to provide some other indicator that the "Staging or Production" section is not quite the same kind of section as the other two, e.g. an interposed sentence, or making both "Inputs" and "Brittleness" be subsections of a "Test Type" section.

4. I propose that we replace the term "Brittleness" with the word "Robustness".

[nejucomo]

Replying to Zancas:

Should nejucomo be given commit access to:

​https://github.com/tahoe-lafs/packaging-tests

and/or:

​https://github.com/tahoe-lafs/buildbot-config-tahoe

?

I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.

[nejucomo]

Replying to Zancas:

...

Zancas and I are chatting in realtime and we both agree that having clear terminology could help avoid confusion.

As an example, in the Inputs section, first bullet, I say "build", but what I means is *not what* python ./setup.py build does.

In fact, all of this description kind of ignores what python ./setup.py build does, which is kind of a "local compilation and install without creating any package".

Other sources of confusion:

"distribution" - does this mean the process of copying bytes from some "package repository" onto a local system and then installing it? *-or-* does it mean "linux distro" or does it mean the format of the bytes in a package or what?

"packaging" - Is this the process of converting source code into a package file (tarball, .whl, .deb, .dmg, etc...)? Is it the process of publishing such files to "package repositories"? Is it the process of installing?

I propose we focus on the checklist in ticket:2049#comment:15 and ignore the description *for now*. Later we can categorize the different tests we create and create a canonical lexicon.

[nejucomo]

Zancas and I are reading over ​buildbot-config-tahoe/tahoe/git/master.cfg in order to learn what should appear in ​packaging-tests.

In my opinion, the master.cfg should be minimal, and I'd like to move some of the operations here into packaging-tests, and here are some examples of why:

• In ​master.cfg line 92 there's a conditional on a per-buildslave option do_test_already_have_dep which determines whether or not to run a test. I would rather have two explicitly named tests such as test_tahoe_build_with_preexisting_pycryptopp versus {{test_tahoe_build_without_preexisting_pyrcryptopp}}} and then configure buildslaves by specifying which tests they run. (In fact, a single buildslave may be able to run both tests which would be handy especially when we have few buildslaves for a given architecture.)

... (I will edit this comment to write more detail, but I want to submit now because I'm paranoid of losing my progress...)

[nejucomo]

Ok, my next goal is to set up a local test buildmaster / buildslave using this config so that I can compare its results to stuff inside ​https://tahoe-lafs.org/buildbot-tahoe-lafs/ before changing anything.

Note: I edited the checklist in comment 15 to reflect this goal.

[daira]

Replying to nejucomo:

Replying to Zancas:

Should nejucomo be given commit access to:

​https://github.com/tahoe-lafs/packaging-tests

and/or:

​https://github.com/tahoe-lafs/buildbot-config-tahoe

?

I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.

I see no reason why the permissions for those two repos shouldn't be identical to those for tahoe-lafs/tahoe-lafs.

[nejucomo]

Replying to daira:

Replying to nejucomo:

Replying to Zancas:

Should nejucomo be given commit access to:

​https://github.com/tahoe-lafs/packaging-tests

and/or:

​https://github.com/tahoe-lafs/buildbot-config-tahoe

?

I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.

I see no reason why the permissions for those two repos shouldn't be identical to those for tahoe-lafs/tahoe-lafs.

Agreed.

[nejucomo]

Note, the new packaging tests, in addition to performing all tests as the previous buildbot configuration should additionally test the OSX packaging added in #182.

Ticket #2303 is about ensuring a buildslave will run this new test.

[nejucomo]

The plan is to add one or more new Builders for packaging tests and not modify existing builders.

[nejucomo]

In a meeting now, we're verbally deciding to bump this out of the OpenITP packaging grant. Instead we will add new packaging tests into tahoe-lafs ./misc/ somewhere and modify the existing buildbot master.cfg for tahoe directly for those tests.

[nejucomo]

Note, I've removed openitp-packaging keyword to reflect our decision.

[meejah]

Ticket retargeted after milestone closed