Packaging

We want to package Tahoe for people to download and use and we want to use 3rd-party libraries. The following are our current desiderata:

• We can produce a Tahoe binary package for a given deployment target, ship that package to a user using that platform, they can install that package, and it will work. That is: user doesn't have to manually satisfy any dependencies. The only deployment target that we currently support is Ubuntu Feisty or newer.

For libraries that Tahoe uses, we have these desiderata:

• We use the source code as it is written by upstream. That is: no patching required.

• Use it as it is packaged by upstream. That is: we prefer to get a copy of the source code of the package as upstream prefers to distribute such source code, rather than by pulling from their revision control tool or so on.

• The user doesn't have to manually resolve any conflicts (this means that we either have to automatically use a 3rd-party library if it is already installed or else we have to automatically force Tahoe to use the copy that it came bundled with).

• Make it convenient for someone to use other versions of the packages that we use e.g. system-wide packages or newer or alternate versions, etc..

• Bundle required libraries with the Tahoe distribution so that if someone downloads the distribution, moves to a desert island without a net connection, and then installs, it works.

setuptools

One option is to use a Python packaging tool named ​setuptools.

Advantages (note that each of these are options provided by setuptools, not requirements imposed by the use of setuptools):

• management of dependencies (even on platforms that don't have a native package manager); This is the important feature.

• integration with ​http://cheeseshop.python.org/ -- with setuptools we can update the ​allmydata-tahoe metadata at the cheeseshop automatically, and also upload packages to be distributed from cheeseshop

• for hackers who want to use Tahoe, and who like setuptools, this makes using Tahoe convenient and pleasant for them

• replace "build/configure/package/distribute/test/develop" code written in the Make language with code written in Python; One specific instance of this is ./setup.py test which runs the unit tests

Disadvantages:

• We haven't figured out exactly how to take advantage of setuptools's dependency management while also integrating smoothly with platform-specific package managers such as Debian apt. I (Zooko) think that this is possible, but it isn't yet standard practice among setuptools users. One important detail is the special option to setuptools called --single-version-externally-managed.

We already have ​a patch which changes Tahoe to use setuptools, but it doesn't solve the "How to integrate smoothly with platform-specific package managers." part yet.

discussion

I came across this page: ​http://wiki.debian.org/DebianPythonFAQ , which suggests that --single-version-externally-managed is the Right Way to build debian packages out of setuptool-using python projects, and that the process is actually fairly easy.

So one way to do this is to have each of our built-in subprojects (allmydata, foolscap, zfec, and now simplejson) produce separate eggs, and have some debian glue that produces separate .debs for each. Given that both foolscap and simplejson have native ways of producing .debs, it would be good for the tahoe .deb to not include them (to allow both tahoe and the native .debs of foolscap or simplejson be installed at the same time).

So we could have a 'build' target (perhaps in the makefile) which produces a handful of eggs, or a handful of .debs, and you need to install whichever ones you don't already have. Or you can just run from source.

-Brian

related tickets

#15 -- consider using setuptools for tahoe

#10 -- clean up after yourself when told to "make clean"

#82 -- remove the "build" step in the "edit, build, run" cycle

#47 -- use pyutil as a separate package and contribute src/allmydata/util/* into pyutil