'make clean' shouldn't remove the src/allmydata_tahoe.egg-info directory

[bertagaz]

When running make clean, the process removes the src/allmydata_tahoe.egg-info directory. This directory is distributed within Tahoe-LAFS sources, which is inconsistent with the fact that make clean should only reset the Tahoe-LAFS directory to a freshly uncompressed state.

This doesn't help in packaging Tahoe-LAFS in distributions like Debian, where tools usually run a make clean before attempting to build. They fail after this first clean, because they detected this directory was removed but is part of the release tarball.

So either make clean shouldn't take care of this directory, or it has to be removed from the release tarball.

[zooko]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/63

[daira]

src/allmydata_tahoe.egg-info is a generated directory that doesn't exist in the checked-in source. Removing review-needed so that this doesn't get committed before I've checked whether this directory should exist in tarballs.

[daira]

A tarball with src/allmydata_tahoe.egg-info removed builds fine and passes tests. So I conclude that this directory should not be in future sdist tarballs. Note that it is standard behaviour of python setup.py sdist to include it, but that is probably a bug in setuptools/zetuptoolz.

[daira]

Alternatively, the Debian etc. packaging should remove the egg-info directory from the source package only. (I emphasize that because Debian packaging of some Python packages has incorrectly tried to avoid installing egg-info directories.)

[zooko]

Either way is fine with me!

[zooko]

Replying to daira:

A tarball with src/allmydata_tahoe.egg-info removed builds fine and passes tests. So I conclude that this directory should not be in future sdist tarballs. Note that it is standard behaviour of python setup.py sdist to include it, but that is probably a bug in setuptools/zetuptoolz.

Actually I disagree with the above. The directory in question is metadata declaring in machine-readable, standard format some facts about the distribution. The fact that it is not necessary to pass tests doesn't mean it should be removed, and I don't see any justification for assuming that its inclusion is a bug.

[daira]

Replying to zooko:

Replying to daira:

A tarball with src/allmydata_tahoe.egg-info removed builds fine and passes tests. So I conclude that this directory should not be in future sdist tarballs. Note that it is standard behaviour of python setup.py sdist to include it, but that is probably a bug in setuptools/zetuptoolz.

Actually I disagree with the above. The directory in question is metadata declaring in machine-readable, standard format some facts about the distribution.

It's regenerated by python setup.py build. We don't include anything else that is generated, apart from src/allmydata/_version.py which is only generated when building from a git checkout.

[zooko]

I don't understand what the issue is here. It sounds like there is a difference of opinion between Debian, which thinks that "make clean" should return the directory to the state it was after it was untarred, and Daira, who thinks that "make clean" should return the directory to the state it was after it was checked out from revision control. Those can't both be true.

[zooko]

Replying to zooko:

I don't understand what the issue is here. It sounds like there is a difference of opinion between Debian, which thinks that "make clean" should return the directory to the state it was after it was untarred, and Daira, who thinks that "make clean" should return the directory to the state it was after it was checked out from revision control. Those can't both be true.

Following-up to myself, the reason those can't both be true is that a source dist is not just a tarball of a source checkout. It also has a version number and (currently) metadata auto-generated.

[zooko]

Another self-follow-up: there are two questions here: 1. should generated stuff (in this case the Python packaging metadata) go into the source dist, and 2. should "make clean" return the directory to the state of a source checkout, or return the directory to the state of a source dist unpack?

The second question is the only one that is blocking Debian, and I don't use "make clean" and I don't care what "make clean" does, so I propose that we make it do what Debian wants it to do, thus answering question 2, and leave question 1 unchanged (in this ticket).

[daira]

Replying to zooko:

I don't understand what the issue is here. It sounds like there is a difference of opinion between Debian, which thinks that "make clean" should return the directory to the state it was after it was untarred, and Daira, who thinks that "make clean" should return the directory to the state it was after it was checked out from revision control. Those can't both be true.

I have no disagreement with Debian's policy here. Debian apparently requires that running make clean after a build should not delete files that were in the source package; I agree with that. (Note that this only partially constrains the answer to your second question in comment:13.) It's also clearly necessary that a source package contains all the files necessary to build -- so it must include src/allmydata/_version.py, but need not include src/allmydata_tahoe.egg_info.

[gdt]

Replying to zooko:

Historically, "make clean" should return to the unpacked-distfile state, and "make distclean" to the revision control system state.

[dstufft]

A Python sdist should contain the .egg-info (or in the future .dist-info). While right now it will technically work without it, there is a push towards making setup.py a build dependency which generates static metadata that is placed in the generated distributions.

I don't know 100% what form this will take but setuptools will be updated to do it automatically so if you start mucking with egg-info it's possible in the future you'll end up with uninstallable distributions, or distributions that require a setup.py fallback.

The long term goal is that the only place setup.py is ever executed is the machine of the person who packages the project.

[daira]

Replying to dstufft:

A Python sdist should contain the .egg-info (or in the future .dist-info).

This assertion is good enough for me; I withdraw my objection to including it. But let's also add make distclean as mentioned by gdt.

[daira]

Replying to daira:

... But let's also add make distclean as mentioned by gdt.

(and then I will never use make clean and will recommend everyone uses make distclean ;-)

[zooko]

I'm fine with adding "make distclean", if it doesn't already exist, and changing "make clean" to not rm the egg-info (as in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/63). Is there anything else we should do at the same time?

[daira]

Merged in [5b55ccc526ac9abf9b2189baeff796e41196fb30/trunk].

[daira]

Please review ​https://github.com/tahoe-lafs/tahoe-lafs/pull/64 for make distclean. (It removes _version.py and _appname.py as well, and I've verified that it is still possible to build and test successfully from a git checkout after this.)

[zooko]

reviewed: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/64#commitcomment-4300256

[daira]

Merged in [90c1b8fb570df5c116a76412e7e4ca55bd5d5a8e/trunk]. Thanks!

[daira]

Updated the comment for make clean: [7f10bf07df8b936e0d62f123072e704bfb8cc963/trunk]