wiki:Packaging

Version 43 (modified by daira, at 2013-05-30T21:40:14Z) (diff)

fix command to build a pycrypto egg; add info about building other eggs

Attention: this page is for Tahoe-LAFS developers. If you are not a Tahoe-LAFS developer, then the page you want is ​quickstart.rst.

see also wiki:Installation, wiki:OSPackages, wiki:Manual, wiki:NewbieDeveloperSetup, and wiki:CompileError

Build Types and Goals

Novice User

In this build type, a user downloads the Tahoe-LAFS source and everything is as automatic as possible. Internet access is assumed, and there is nothing wrong with using it.

Developer

In this build type, someone would like to make changes to Tahoe-LAFS, and gets source from git. TODO

Packaging System

In this build type, the build user is a packager of rpms/debs or pkgsrc. Rather than convenience, the concern is a stable and repeatable build that produces exactly the same output bits regardless of which prereqs were already installed. Using the network is not acceptable.

Within a packaging system, there are typically build phases, loosely derived from how autoconf works.

  • fetch: obtain the "distfiles" (tarballs published by e.g. the tahoe project)
  • patch: apply OS-local/pkgsrc-local changes (goal is to drive these to zero)
  • configure: search for dependencies and configure paths
  • build: compile sources to object form, run nroff to make man pages, etc.
  • install: place bits either in the running system or in a destdir, suitable for running
  • package: create a binary package from the installed bits

quickstart.rst

The setup method that we offer to people when we don't know what platform they are setting up on or what level of expertise they have is "the quickstart.rst method". This method starts by telling them to go to ​http://tahoe-lafs.org and follow "the quickstart instructions" link.

  • quickstart.rst is intended to be as platform-independent as possible.
  • quickstart.rst should not instruct the user to install any dependency if the build system could instead satisfy that dependency automatically (currently, as of 2011-08-16, the only dependency that can't be satisfied automatically for a build from trunk is Python).
  • quickstart.rst is intended to be as simple as possible. Options (you can do it this way or that way as you prefer) or branches (if you are in this situation do this, else do that) are strongly discouraged from being written into quickstart.rst. If at all possible, please move those to AdvancedInstall.

The goal is to start with a bare OS and a network connection (no Python, no compiler) and get to a working ./bin/tahoe by downloading a few things and running setup.py build (which downloads the rest).

binary .egg's of dependency native-code packages

For quickstart.rst to work on your platform there must exist a binary .egg for your platform of each dependency that has a native-code extension module, which are listed in the columns of ​this table. If there is no binary .egg available of one of those packages for a given platform then this is a bug which prevents quickstart.rst from working on that platform. (Note: if you already had that Python package installed on your system then you would be able to setup Tahoe-LAFS even if there is no binary egg of that Python package available, e.g. if you manually installed the package already or if you installed the package already using an operating system packaging tool like apt-get. Nonetheless, since we can't rely on the user having done that and since quickstart.rst cannot instruct the user to do that -- see the section about quickstart.rst above -- then the absence of a binary .egg of a dependency for a platform is a bug preventing quickstart.rst from working on that platform.)

The eggs are built from the corresponding source packages:

Except for PyCrypto (see below), each egg is built by running python setup.py bdist_egg in the root of its source distribution.

Creating a binary egg for PyCrypto

PyCrypto's setup.py does not support setuptools, so the bdist_egg target is not available out of the box. To build an egg, use the following command:

$ python -c "import setuptools; __file__ = 'setup.py'; execfile('setup.py')" bdist_egg

Detailed instructions for building a PyCrypto egg on Windows: HowtoBuildPyCryptoOnWindows.

AdvancedInstall

An alternative method to quickstart.rst is AdvancedInstall. This method would be appropriate for people who prefer for the setup to be done differently than the way quickstart.rst works. It is also the fallback for people who have tried quickstart.rst and it didn't work (although the fact that quickstart.rst didn't work should also be treated as a build process bug even if falling back to AdvancedInstall allows the user to proceed).

Packaged by your operating system

See wiki:OSPackages

Desiderata

We want to package Tahoe-LAFS for people to download and use and we want to use 3rd-party libraries. The following are our current desiderata:

  • For certain deployment targets (namely, recent Debian and recent or Long-Term Support Ubuntu), our Debian packagers produce a Tahoe-LAFS binary package for that platform, and upload it to the standard apt repositories. Our users can install that package by running 
      sudo apt-get install tahoe-lafs
    and it will Just Work. That is, users don't have to manually satisfy any dependencies by building other packages.

For libraries that Tahoe-LAFS uses, we have these desiderata:

  • We use the source code as it is written by upstream. That is: no patching required.
  • Use it as it is packaged by upstream. That is: we prefer to get a copy of the source code of the package as upstream prefers to distribute such source code, rather than by pulling from their revision control tool or so on. (But we strongly prefer source packages to binary.)
  • The user doesn't have to manually resolve any conflicts (this means that we either have to automatically use a 3rd-party library if it is already installed or else we have to automatically force Tahoe-LAFS to use the copy that it came bundled with).
  • Make it convenient for someone to use other versions of the packages that we use e.g. system-wide packages or newer or alternate versions, etc..
  • Bundle required libraries with the Tahoe-LAFS distribution so that if someone downloads the distribution, moves to a desert island without a net connection, and then installs, it works.

setuptools

We use a Python packaging tool named ​setuptools (actually a fork of it named ​zetuptoolz).

Advantages:

  • management of dependencies (even on platforms that don't have a native package manager); This is the important feature.
  • for hackers who like setuptools, this makes using Tahoe convenient and pleasant for them
  • it allows "build/configure/package/distribute/test/develop" code written in the Make language to be replaced with code written in Python. One specific instance of this is ./setup.py test which runs the unit tests. (We still have a makefile, but it is possible to build and develop Tahoe-LAFS without using it.)

Disadvantages:

  • setuptools has many bugs and is poorly maintained.
  • these bugs affect Tahoe-LAFS disproportionately because it has more, and more complicated, dependencies than most Python packages. We currently have 6 open setuptools-related tickets (full list) and have closed 104 such tickets (full list). (These counts may be a bit high because of duplicates and non-bugs, but they're indicative of the problem.)
  • maintaining our fork (which is necessary to avoid some of the bugs) takes up developer time and mental bandwidth that would be more productively spent on Tahoe-LAFS itself.
  • even when working as designed, setuptools has some serious problems. For example, people who package Tahoe for OS distributions generally dislike very much its behaviour of automatically downloading dependencies from random websites (however convenient something like that would be if it were optional and designed properly).