[tahoe-dev] buildbot update

[Brian Warner]

On Mon, 9 Mar 2009 09:54:00 -0700
Shawn Willden <shawn-tahoe at willden.org> wrote:

> On Monday 09 March 2009 10:24:58 am zooko wrote:
> > The Debian package builder on Shawn's intrepid box is successfully
> > building .debs, but has nowhere to upload them.  Brian, Shawn: could
> > we arrange it so that it uploads the .debs to a directory on a Tahoe
> > grid?  :-)
> 
> There's a Tahoe node that usually, but not always, runs on the intrepid
> box, connected to the test grid. Probably a better choice is the test grid
> node that is running on a neighboring machine (IP 10.0.1.1, port 3456).

For this purpose, we might want to upload them to the allmydata prodgrid
instead.. might be more reliable and less stall-sensitive. I dunno.

For the record, here's our overall plan:

 * create a master tahoe directory structure, in the shape of a regular APT
   repository
   (dists/{gutsy,hardy,intrepid}/tahoe/{source,binary-amd64,binary-i386}/*.deb)
 * extract the writecap for dists/intrepid/tahoe/binary-amd64 , give it to
   Shawn, he writes it into a file accessible by his buildslave (maybe by doing
   'tahoe set-alias tahoe-intrepid-amd64-debs WRITECAP')
  * thanks to tahoe's simple capability-based access control, Shawn (and
    his buildslave) get the ability to publish intrepid-amd64 debs but not
    the ability to tamper with anything else. Magic!
 * configure the build process to use 'tahoe put *.deb
   tahoe-intrepid-amd64-debs:' as its upload step, then invoke some
   as-yet-undefined mechanism to notify the "APT repo manager" to update the
   index

 * on the server side:
  * build an APT repo manager: a daemon that sits around waiting to uploaders
    to tell it that a new file has been added to the Tahoe directory
    structure. When that happens:
   * download the structure to local disk. This could just use "tahoe cp -r
     repo: ./repo", but we'd prefer it to have a sort of inverse backupdb:
     something that lets us refrain from doing a download when the local file
     looks sufficiently up-to-date.
   * run the APT index-building scripts to create Packages.gz, Contents, etc
   * mirror everything back into tahoe: 'tahoe cp -r ./repo repo:' . Again,
     this would be better if 'tahoe cp' used the backupdb, but this is an
     easier feature addition than the downloaddb ('restoredb'?) described
     above
   * mirror everything over to http://allmydata.org/debian
  * publish the tahoe readcap for that 'repo:' directory

With that in place, there will be two ways to make your debian box have
bleeding-edge tahoe packages: either point APT at allmydata.org, or point APT
at a tahoe webapi server and the published readcap. The latter is slightly
more self-hosting and cooler, but if the testgrid isn't being reliable, you
might prefer to use a normal HTTP site instead.

The "APT repo manager" notification step might be a tiny Foolscap client..
there's a demo of something similar in the foolscap docs/ directory already.

This week is looking kind of busy, I'm not going to have much time to work on
this project, but maybe next week. I'll let you know when/if I build any of
the pieces listed above. Note that we could set up the buildslave to upload
.debs to a directory independently of me getting the time to change the APT
repo manager scheme around (the debs just wouldn't be APT-gettable for a
while, people could still fetch them manually from the published readcap).


cheers,
 -Brian