init scripts included in the apt-get install for tahoe-lafs on Debian/Ubuntu derivatives

[stott]

/etc/init scripts should be included by default to start tahoe-lafs if there are configured nodes, introducrs or helpers, Tahoe should auto-start. Run level 3, after most other services are up and running.

[USSJoin]

So I just spent a few hours trying to write an upstart (the new ubuntu replacement for init.d) script for Tahoe. It appears not to be possible, because the Tahoe daemon forks in such a way that the initctl processes can't follow it (with either the "expect daemon" or "expect fork" stanzas).

However, there's an easy workaround I've found, that I'm now using:

ln -s /usr/local/bin/tahoe /etc/init.d/tahoe

This *does* require that the "root" user be able to run tahoe, which requires an

ln -s /path/to/your/.tahoe /root/.tahoe

But that's also required by any (theoretical) upstart script.

[davidsarah]

Replying to USSJoin:

However, there's an easy workaround I've found, that I'm now using:

ln -s /usr/local/bin/tahoe /etc/init.d/tahoe

This *does* require that the "root" user be able to run tahoe, which requires an

ln -s /path/to/your/.tahoe /root/.tahoe

But that's also required by any (theoretical) upstart script.

Note that this conflicts with #725 ("we should whine if we're running as root"). Apart from being a really bad idea because of excess privilege, running the gateway as root causes directories and files to be owned by root and not accessible to other users, which then hoses any attempt to run the gateway as another user. (The CLI should still work provided that tahoe create-node or tahoe create-client wasn't run as root.)

[ioerror]

An init.d script created for Debian Lenny (from /etc/init.d/skel)

[ioerror]

this file goes in /etc/default/tahoelafsd

[ioerror]

A Debian user should do the following to securely run Tahoe:

addgroup --system tahoelafsd
adduser --disabled-login --system --home /var/lib/tahoelafs --shell /bin/bash --ingroup tahoelafsd tahoelafsd 
tahoe create client /var/lib/tahoelafs/

When using the init.d script that I have attached, Tahoe will run as the user tahoelafsd user and not as root.

The Tahoe packages should include this as part of the files provided in debian/

[ioerror]

It's probably a good idea to keep the entire tahoelafs home directory hidden from the rest of the system:

chmod 750 /var/lib/tahoelafs/

It's important to note that you'll want to edit the /var/lib/tahoelafs/tahoe.cfg file after the creation of the client config or the init.d script will not work.

By default, I think that the init.d script should simply not start tahoe without a configuration... I'll add that and replace the DEFAULTS file....

[ioerror]

New defaults file for debian

[ioerror]

tahoe defaults file

[ioerror]

finial version of init.d script that uses defaults properly

[ioerror]

It's likely that Tahoe (post-configuration) should start at boot:

# update-rc.d -f tahoelafsd defaults
 Adding system startup for /etc/init.d/tahoelafsd ...
   /etc/rc0.d/K20tahoelafsd -> ../init.d/tahoelafsd
   /etc/rc1.d/K20tahoelafsd -> ../init.d/tahoelafsd
   /etc/rc6.d/K20tahoelafsd -> ../init.d/tahoelafsd
   /etc/rc2.d/S20tahoelafsd -> ../init.d/tahoelafsd
   /etc/rc3.d/S20tahoelafsd -> ../init.d/tahoelafsd
   /etc/rc4.d/S20tahoelafsd -> ../init.d/tahoelafsd
   /etc/rc5.d/S20tahoelafsd -> ../init.d/tahoelafsd

[ioerror]

the lenny files required to install, remove, and purge a .deb

[ioerror]

With the above patch, I've installed, uninstalled, reinstalled, purged, and then installed again. I have configured my daemon to run by editing /etc/defaults/allmydata-tahoe and having a configured /var/lib/tahoelafsd directory...

I've also configured it to run at boot like so:

 update-rc.d -f allmydata-tahoe defaults
 Adding system startup for /etc/init.d/allmydata-tahoe ...
   /etc/rc0.d/K20allmydata-tahoe -> ../init.d/allmydata-tahoe
   /etc/rc1.d/K20allmydata-tahoe -> ../init.d/allmydata-tahoe
   /etc/rc6.d/K20allmydata-tahoe -> ../init.d/allmydata-tahoe
   /etc/rc2.d/S20allmydata-tahoe -> ../init.d/allmydata-tahoe
   /etc/rc3.d/S20allmydata-tahoe -> ../init.d/allmydata-tahoe
   /etc/rc4.d/S20allmydata-tahoe -> ../init.d/allmydata-tahoe
   /etc/rc5.d/S20allmydata-tahoe -> ../init.d/allmydata-tahoe

[davidsarah]

Needs documentation. Also, is dpkg --remove supposed to remove the user/group?

The ​Debian FAQ says:

• Remove a package (but not its configuration files): dpkg --remove foo
• Remove a package (including its configuration files): dpkg --purge foo

(also see ​here).

Maybe --purge should remove the user/group and --remove should not (but I'm not an expert on Debian package management).

[davidsarah]

Replying to davidsarah:

Also, is dpkg --remove supposed to remove the user/group?

Oh, you changed that in a later patch. Never mind, that looks right now (remove only the user on purge).

[francois]

This patch fix a few missing commands in ioerror's modifications to docs/debian.txt

[ioerror]

Only two packages aren't .debs now

[ioerror]

What other documentation should I write up? Perhaps a man page? Or just continuing to flesh out the debian.txt?

I've attached my current debian.txt - it's how I build Tahoe on my current production system that is on the volunteer grid. I changed a few steps to match the documentation patch from francois.

[ioerror]

There's one issue that is currently outstanding in this package but seemingly unrelated to Debian. It appears that setuptools has a multiple input warning:

# /etc/init.d/allmydata-tahoe start
/usr/lib/python2.5/site-packages/zope/__init__.py:19: UserWarning: Module allmydata was already imported from /usr/lib/python2.5/site-packages/allmydata/__init__.py, but /usr/lib/python2.5/site-packages is being added to sys.path
  import pkg_resources
STARTING /var/lib/tahoelafsd
client node probably started

[ioerror]

diff -u allmydata-tahoe /tmp/allmydata-tahoe.init 
--- allmydata-tahoe.init     2010-03-04 00:48:45.000000000 -0800
+++ /tmp/allmydata-tahoe.init    2010-03-05 19:31:53.000000000 -0800
@@ -21,7 +21,6 @@
 DAEMONHOME="/var/lib/tahoelafsd/"
 DAEMON=/usr/bin/$NAME
 DAEMON_ARGS=" start $DAEMONHOME"
-PIDFILE=/var/lib/tahoelafsd/twisted.pid
 USERNAME=tahoelafsd
 SCRIPTNAME=/etc/init.d/$NAME

[zooko]

Let's see... this ticket was "assigned" to warner (by me) but he never "accepted" it, so he hasn't indicated that he actually intends to review it soon. Could someone else review it?

[warner]

FYI, "tahoe start" is a simple frontend to "twistd", the Twisted daemon-launching tool. Eventually we'll fix it so that any extra arguments you pass to "tahoe start" will be copied to "twistd", which will let you get a bit more control over how the daemon is started.

In the meantime, you can run twistd directly. "tahoe start BASEDIR" is equivalent to something like "cd BASEDIR && twistd -y *.tac --logfile logs/twistd.log". If you add --nodaemon to the twistd arguments, it won't daemonize. If you want to launch tahoe from an init.d script using start-stop-daemon, that's probably what you want to use. You might also want to use the --pidfile argument to control where the pidfile goes.

[zooko]

What's the status of this ticket? It was assigned to Brian but he hasn't accepted it, and it is marked as review-needed. I don't understand if Brian's comment:16 was intended as a code review requesting further changes or if it was just helpful commentary. I'm unassigning this from Brian and issuing a call for a reviewer. PatchReviewProcess

[zooko]

Brian: I wonder if people are put off from taking another step on this ticket because they aren't sure what you meant in comment:16. Just in case, would you please state whether the code here is good enough or whether in your opinion changes are needed? If the former, please remove review-needed from the keywords and add reviewed to the keywords. If the latter, please remove review-needed from the keywords. Thanks!

[taral]

python-foolscap is packaged in debian sid.

Instead of packaging argparse and zbase32, you could just include them in the allmydata package directly. They are pretty small.

+1 to running twistd directly instead of using tahoe start. Or add a --nodaemon flag to tahoe start that causes it to exec twistd instead of forking?

Why do you have automatically generated sections in your postinst? Oh, I see. darcs unrecord is your friend.

Why not remove the group you created?

[warner]

Zooko: comment:16 was intended as helpful commentary. Folks who want to write init scripts but who are put off by "tahoe start"'s automatic daemonizing. Such folks could use the information in comment:16 to write a better init script.

I haven't touched tahoe+debian packaging in a long time.. I'm not currently familiar with how we're doing it (if at all). The last I remember was planning a new out-of-tree scheme using git-buildpackage and a collection of schroot jails (of various ubuntu/debian) versions on an EC2 host. My thought was to copy the debian.diff files from that effort back into the tahoe source tree. I vaguely remember thinking that I wanted to keep the in-tree stuff simple, and leave things like init.d scripts "to the professionals" (i.e. the actual ubuntu/debian packages that ship in Lenny, etc), and perhaps as a result not putting energy into landing tese patches.

So I cannot currently sign off on the patches: I don't have the time to page that stuff back into my brain and do a proper review. Maybe next week.

[davidsarah]

I had reviewed a previous version of these patches, and the changes since then look good. I think that any remaining issues (e.g. comment:13) are minor and new tickets should be opened for them.

I suggest committing these patches for 1.7.1, or failing that 1.8beta. They won't cause regressions; the worst that can happen is that the scripts don't do precisely what users want.

[zooko]

applied in ca660a5fc6cb8d55, babbdf949437116b. Thanks, Jacob and David-Sarah!

[zooko]

I was talking with intrigeri about including Tahoe-LAFS in Tails (​https://tails.boum.org/), and he pointed out this Debian bug report: ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652003 . That bug report by Thomas Pierson <contact@…> says that he can't find the actual init script anywhere. Sure enough, reviewing this ticket it appears to me that we accidentally omitted the actual init script (final version attached to this ticket by ioerror: attachment:debian-init-script-patch-final​).

Daira: did you review attachment:debian-init-script-patch-final​? You wrote review notes in comment:22.

ioerror: why did you post the diff in comment:14? Why remove the PID file variable setting?

[gdt]

Is there a documented general strategy for how to place OS-specific helper files? It seems that these could be provided for a dozen systems, at least and thus there should be some subdirectory structure to avoid this getting cluttered.

There's also the issue of whether things are included with the sources or are part of packaging. Probably most things belong in the sources so they can be used/tested independently of packages. But user creation feels like it should be a packaging thing.

[daira]

Replying to zooko:

Daira: did you review attachment:debian-init-script-patch-final​?

Yes (and I just rereviewed it). I'm not an expert on Debian packaging, but it looks fine to me, modulo this:

ioerror: why did you post the diff in comment:14? Why remove the PID file variable setting?

I'm not ioerror, but the PIDFILE variable appears to be unused (and the pid file is called twistd.pid anyway, not twisted.pid).

[daira]

Replying to daira:

Replying to zooko:

Daira: did you review attachment:debian-init-script-patch-final​?

Yes (and I just rereviewed it). I'm not an expert on Debian packaging, but it looks fine to me, modulo this: [...]

and misc/lenny no longer being an appropriate directory to put it in.

[zooko]

Per ticket #1454, we decided to remove the debian package files from the upstream source repository since those files were now being maintained by debian. This turns out to be partially mistaken, as this particular set of files (the init scripts), we never successfully transfered to the Debian maintainers yet!

[zooko]

Okay I created a branch in github, but it should not be merged to trunk because of #1454. As I said in the commit message, I'm creating this branch solely to give Debian/Ubuntu/Tails packagers and us a shared codebase to point at and talk about:

​https://github.com/zooko/tahoe-lafs/commit/cb6d458ab5c638f7b203ef07b91b4fb5873d91e0

[zooko]

Okay, I've posted to ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652003 asking the Debian maintainers to take over the init scripts. Closing this ticket as "Somebody Else's Problem".

[zooko]

I'm leaving this ticket closed, but here I'm posting a copy of my code-review of bertagaz's patch which is now in Debian.

Thank you for working on this! I just reviewed ​http://anonscm.debian.org/gitweb/?p=tahoe/tahoe.git;a=blob;f=debian/README.Debian;h=14ea2a7c1b38d41df36dd052c44cdf22603fd775;hb=87f7666c2c3a5059dc28ea95c336b9de7f08ae47 . It is interesting that this is using the username within the local operating system as the "nick". That isn't what we intended the nickname to be for, but I don't see anything obviously wrong with it. It means that the node will announce to everyone who connects to it what the local username is. Is that what was expected?

Hm, actually now that I think this through, I think the human reading that doc needs a warning about this! I think you should add a sentence that says "This nickname will be announced to all participants in the grid." or something like that. Or, change the suggested command-lines so that it no longer fills in the "nick" field from the username.

[zooko]

Okay, I just reviewed ​http://anonscm.debian.org/gitweb/?p=tahoe/tahoe.git;a=blob;f=debian/tahoe-lafs.init;h=13b505fb4b2d4be959e3df7edef02a369a48fc7c;hb=8428876521454b5fd2b0719048caf909c0ab68ee . Here are my comments.

Thank you for working on this patch! I'm excited about making Tahoe-LAFS be a more first-class citizen of the Debian universe. A lot of good can come of this. Thank you for your contribution.

I don't understand why /etc/init.d/tahoe-lafs restart does tahoe stop ; sleep 1 ; tahoe start and /etc/init.d/tahoe-lafs force-reload does tahoe restart. I think both of those should do tahoe restart.

I think /etc/init.d/tahoe-lafs stop should do tahoe stop instead of kill `cat twistd.pid`. That's because tahoe stop has a couple of features such as warning the user if the daemon doesn't stop after SIGKILL. See the code here.

Other than those two things, I didn't see anything strange or objectionable in this patch. Thanks again!

[zooko]

I posted another, much bigger, review to ​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652003 but I'm not copying it into this ticket, so go read that thread.