revamp "tahoe run"

[meejah]

In the past, "tahoe run" was based around the (old) Twisted daemonization / runner command called "twistd".

As part of this, "twistd.pid" files are written out. Another consequence is that twistd's options are also available to "tahoe run".

We have since removed the ability of "tahoe run" to daemonize itself -- instead users are encouraged to use their favourite damonization software.

We would like to change how the PID-files work a little to match magic-folder and to give any "parent process" a better chance of determining if the pid-file is stale.

This means adding the creation-time of the process. That is, the pid-file will look like PID create-time\n. The creation-time helps in case a process-id is recycled: it is then possible to determine if that process is tahoe (because the creation-time will match) or something else (the creation time will be newer).

We could simply write _another_ PID-file with the above information, however it seems like now might be a good time to get rid of the twistd dependency/use entirely.

However, that's technically a backwards-incompatible change because tahoe-run currently takes all of the twistd top-level options....it's _possible_ some of these are actually useful, but a lot of them seem pretty "twistd"-specific, and I'm not sure what actually happens if you specify any of them.

I tried tahoe run testgrid/bob --profile foo.profile for example, but did not get a "foo.profile" file (which is I assume how that option is supposed to work).

[exarkun]

It seems like refactoring away from twistd and adding the creation time information should be separate tasks. It makes sense that implementing the creation time recording might be simpler without having to worry about the specific factoring twistd imposes on us so I can see the motivation to do the refactoring first. I'd still try to land that independently of any behavior changes.

As far as what functionality we're comfortable dropping from twistd, the following options seem useful and it seems reasonable that we'd want to still have them (or equivalents) with our new command-line runner:

• debug
• reactor
• help-reactors
• profile
• profiler
• savestats
• pidfile (or equivalent for "pid and creation time file")

Profiling does work via twistd with the tahoe cli but the UI is wonky: the default profiler is the bad one so you always want --profiler cprofile; by default the text report is written, ordered by _name_, which is almost never what you want, so you almost certainly also want --savestats; and tahoe changes its working directory to the node directory before profiling is set up so the path you pass to --profile is interpreted relative to that directory (which I guess is why you didn't see a profile report written where you were looking).

These are tempting but not really critical:

• rundir
• uid, gid, euid
• umask

These all have to do with process-global state that you often want to control in a careful deployment - but they're also all things that whatever process launches tahoe can initialize so at best they would be a convenience for folks without tools to control them some other way (which is probably not anyone doing a careful deployment).

Assuming we're going to keep going with "just write all your logs to stdout", we don't need:

• logfile
• logger
• syslog
• prefix

(Presumably we keep *our* options for configuring the eliot log though.)

These are silly and Tahoe certainly doesn't need them:

• chroot
• encrypted
• file
• nodaemon
• no_save
• originalname
• source
• spew
• python

As far as backwards compatibility, the path might be to not do the refactoring in-place but instead just implement the new cli (maybe just new subcommand) and provide it along-side the existing one. I think the main downside of this approach is it requires thinking of a new name. Aside from that, it has good properties: it lets us deprecate the old cli over whatever timeline we like while giving early adopters the new stuff right away; if there's a problem with the new stuff in some environment, people can always stay on the old stuff while they wait for fixes, etc. This also lets us develop and release the work incrementally. We don't have to finish the _whole_ replacement CLI before landing _any_ of it in master. That's probably good since this looks like a big chunk of code, even with the fairly limited set of options.

Given all of that, I wonder if -- for the sake of expediency -- we want to implement the pid-and-creation-time file feature within the current framework first. Otherwise it seems like we would be looking at quite a bit of development effort before that feature becomes usable by the first anticipated audience (GridSync?).

[meejah]

It seems like this may imply a "want" to have a more-general CLI re-vamp.

@exarkun and I discussed elsewhere that starting fresh may be a good idea, that is something like lafs * becomes a place to have a more-thought-through API (but we mark it as "experimental" or at least "not-yet-stable" for a few releases).

An similar option would be something like tahoe run2 for "re-thought" commands which eventually get promoted to tahoe run or similar.

That said, this is well beyond the scope of the feature I was trying to add so I don't have _immediate_ plans to tackle any of this. Collecting ideas here is good though.

[hacklschorsch]

Isn't the pidfile mtime sufficient to see if its stale?